Machine versus Human Attention in Deep Reinforcement Learning Tasks
Authors: Sihang Guo
Ruohan Zhang
Bo Liu
Yifeng Zhu
Dana Ballard
Mary Hayhoe
Peter Stone
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Interpretability
Attention
Eye Tracking

Abstract: Deep reinforcement learning (RL) algorithms are powerful tools for solving visuomotor decision tasks. However, the trained models are often difficult to interpret, because they are represented as end-to-end deep neural networks.  In this paper, we shed light on the inner workings of such trained models by analyzing the pixels that they attend to during task execution, and comparing them with the pixels attended to by humans executing the same tasks. To this end, we investigate the following two questions that, to the best of our knowledge, have not been previously studied. 1) How similar are the visual representations learned by RL agents and humans when performing the same task? and, 2) How do similarities and differences in these learned representations explain RL agents' performance on these tasks? Specifically, we compare the saliency maps of RL agents against visual attention models of human experts when learning to play Atari games. Further, we analyze how hyperparameters of the deep RL algorithm affect the learned representations and saliency maps of the trained agents. The insights provided have the potential to inform novel algorithms for closing the performance gap between human experts and RL agents.

Machine versus Human Attention in Deep
Reinforcement Learning Tasks
Sihang Guo UT Austin sguo190utexas.co edu
Ruohan Zhang Stanford University zharu@stanf ord. edu
Bo Liu UT Austin bliuêcs utexas.edu
Yifeng Zhu
UT Austin yifeng zhuuutexas edu,
Dana Ballard
Mary Hayhoe
Peter Stone
UT Austin danablutexas.cr edu,
UT Austin
UT Austin, Sony Al
Sayhoe@utexxs.eddu
pstoneêcs wtw.asteee... edu,
Abstract
Deep reinforcement learning (RL) algorithms are powerful tools for solving visuo- motor decision tasks. However, the trained models are often difficult interpret, because they are represented as end-to-ent deep neural networks. In this paper, we shed light on the inner workings such trained models by analyzing the pixels that they attend to during task execution, and comparing them with the pixels attended by humans executing the same tasks. To this end, we investigate the following two questions that, to the best of our knowledge, have not been previously studied. 1) How similar are the visual representations learned by RL agents and humans when performing the same task? and, 2) How do similarities and differences in these learned representations explain RL agents' performance on these tasks?
Specifically, we compare the saliency maps of RL agents against visual attention models human experts when learning to play Atari games. Further, we analyze how hyperparameters of the deep RL algorithm affect the learned representations and saliency maps of the trained agents. The insights provided have the potential to inform novel algorithms for closing the performance gap between human experts and RL agents.
Introduction
Researchers have devoted much effort understanding deep neural networks (DNNs). Since DNNs are partially inspired by the biological nervous systems, researchers have compared these neural networks with the human brain and sensory systems by asking two typical types of questions. The first one considers representation: How similar are the visual representations learned by DNNs and humans when performing the same tasks? The second similarities and differences in the learned representations explain DNNs' performance on their tasks?
concerns explainability: How do
The first question motivates seminal line of research that compares representations learned by DNNs with those Iearned by humans. In computer vision, a linear model that uses the feature map activations generated by trained CNN accurately predict neural activities in the early visual cortex, indicating that the two systems have learned similar visual representations [19] 78 79). In language learning, similar connection is found between deep language models and cortical areas [1] 34).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

However, this type of comparison has just emerged in Decision-making research 44 [3), which motivates us compare DNNs trained for deep reinforcement learning (RL) tasks with human decision-making. We ask: Do deep RI. agents and humans agree on what visual features are important? In other words, do agents pay attention to the same visual features as humans do?
The explainability question is motivated by the Explainable AI (XAI) for deep RL agents [29] 55 Deep RL has achieved many successes, but few of us fully understand these agents. These agents often learn a mapping from raw images to actions end-to-end where it is not clear why particular decision is made. Our work addresses the explainability of these black-box models: Do RL agents make mistakes because they fail to attend to important visual features that matter for making the correct decision? An expert human's 's attention data could serve as a useful reference for identifying important visual features, which has been validated in object recognition tasks [50].
To answer these two questions, we situate our research in the domain of Atari games 6 These games span a variety of dynamics, visual features, reward mechanisms, and difficulty levels for both humans and AIs. Two recent lines of research have made our work possible. On the one hand, human attention dataset on Atari games has been collected using eye trackers [85] and convolution- deconvolution networks have been shown to accurately predict human attention distributions [83] On the other hand, tools have been introduced to generate visual interpretations of RL agents [23] so one can visualize an RL agent's saliency map given an image, which is a topographically arranged map that assigns an importance weight to each image pixel and is often treated as the "attention map" of the RL agent. However, these tools can only provide qualitative results we still need to manually inspect the saliency maps to interpret the performance. A useful comparison standard like human attention had not been considered hence systematic quantification was impossible.
In this work, we present a novel study that compares the RL agent's ' attention with both real and estimated human attention. We analyze how learning and hyperparameters of the RL algorithm affect the learned representations and saliency maps. We analyze failure and unseen states for RL agents and identify potential challenges to achieve human-level performance. We conclude by discussing insights gained for RL researchers in both cognitive science and AI.
2 Related Work
Human vs. machine attention in vision and language tasks. Human experts' gaze is very efficient and accurate for solving vision tasks. The peak angular speed of the human eye during a saccade reaches up to 900 degrees per second [59]. This allows humans to move their foveae to the right place at the right time to attend to important features [17. Therefore, human expert's gaze serves good standard in many vision-related tasks for evaluating machine attention, as a learning target for training machine attention
84] This approach widely used in computer vision, see Nguyen et al. [51] for review. For example, visual saliency researchers train DNNs to predict human visual attention. Saliency research is a well-developed field and we direct interested readers to recent review papers (7 10 26 9. One such paper has compared visual saliency models with human visual attention [39]. In vision-related language tasks, such as image captioning and visual question answering, it found that the saliency maps of DNN models are different from human attention [25 27). Understanding and quantifying such differences have provided insights on the performance, especially in failure scenarios, of these vision-language models.
Visual explanation for deep RL. Two classes of methods are widely used to generate visual interpretations of DNNs in the form of saliency maps: gradient-bused and perfurbation-bused. Gradient- based methods compute saliency maps by estimating the input features' influence on the
output using the gradient information [67 68
43
82 66,
71
64 12 86). These methods are for
visualizing general DNNs but have been used to interpret deep RL agents [38 76 65 35 75). Yet gradient-based saliency maps often lack physical meaning and could be difficult to interpret-ther may thus are not used in our analyses.
highlight regions with no task- relk-relevant objects or features 23 25,,
Perturbation- based methods alter parts of the input image and measure how much the output is affected by the change. Hence there have been different methods of altering the input [8] 22 14
87 58
These methods have been applied to Atari deep RI. agents and can generate qualitatively meaningful
saliency maps 23 33 24 [6]. However, without human attention as a reference, it difficult to
quantitatively analyze these saliency maps, which motivates our work. There are also methods that

---

change the architecture of the deep RL network by augmenting it with an explicit artificial attention
module, so that one can directly access its attention map 449 80 48
Researchers have taken this
approach and compared RI. agents" attention with human attention [52] However, these methods do not apply to general deep RI. algorithms since they need to modify the original network architectures and retrain the new ones.
3 Method
We now discuss methods for modeling human attention, training RI. agents, and extracting attention information from the trained RL agents. We then discuss how we select data for comparing human versus RL attention, and define the comparison metrics.
Human attention data and model We use human expert gaze data from Atari-HEAD dataset 85 The original game runs continually at 60Hz (6. speed that is challenging even for professional gamers. Human eye movements were rushed and inaccurate at this speed and hence could not serve as a useful reference. In [85], however, the human data were collected in semi-frame-be frame-by--- boane-by-frame mode. Without changing the action manipulation or the reward structures of the games, this design mimicked the frame-by-frame processing of RL agents for the human players. The players were allowed enough time to process the visual input, producing higher-quality data with more precise attention allocation and action execution. These advantages led to world expert-level performance in 85 about 35x better than the human performance reported in previous deep RL literature [46] 28
To get human saliency maps for the image states generated by RL agents, need an accurate human attention model. RL agents make many more mistakes than expert humans and cannot reach the late stages of the games. This creates a state distribution that does not match human data, which contains mostly good states. Having a human attention model allows us to perform comparisons in states encountered by RL agents only, which is especially important when we later analyze RL agents' failure states.
Predicting human attention is challenging task in Atari games. The human gaze is rarely on the player's is avatar or contingency regions [6] so simple object detectors would not work. Human players often select and focus few among multiple visually identical objects, and divide their attention if multiple objects are relevant for decision making. Recently, variants of convolution-deccomolluu network models have achieved the best results on predicting human attention 42 83 53
[6]. Here we
followed their approach and trained gaze models from ground truth human gaze data. Given an image, the models produce heatmap of the same dimension that predicts human attention distribution. These models have been evaluated and described in detail in previous studies (c.g.. 18),, so we leave out the details in Appendix As a control analysis, we include two attention models in addition to the human gaze. The first one captures the motion information, measured by Farneback optical flow between two consecutive images 20) The second model captures salient low-level image features, including color, orientation, and intensity (weighted equally), computed by the classic Koch saliency model 32 Implementation details of these models can be found in Appendix 1.
Reinforcement learning agent and attention model As a case study, we use a popular deep RL algorithm named Proximal Policy Optimization (PPO) [3] with default hyperparameters [30]. For each experimental condition (discussed later), we train 5 models with different random seeds to capture the variance in training and ensure reproducibility. For the Atari gaming environment, we use the basic version that has no frame skipping and no stochasticity in action execution (NoFrameskip-v4 version). In order to capture variance in training and ensure reproducibility, we select six popular Atari games instead of using all Atari games and train 540 agents in total, with the same hyperpareameters except for random seeds and discount factors (see Section 4 300 GPU days on GeForce GTX 1080/1080 Ti).
We use a perturbation- based method 2 to extract attention maps from PPO agents, which has been validated by several subsequent studies 25 [5] The algorithm takes an input image I and applies a Gaussian filter to a pixel location (i. j) to blur the image. This manipulation adds spatial uncertainty to the region around and produces a perturbed image $ i.j). Denote the learned policy as л and the inputs to the final softmax activation as TT(T) for image (i.e. the last latent representation). saliency score for this pixel (i,j) can be defined as how much the blurred image changes the latent
3

---

representation Tu (I) in Euclidean space:
= [_
(x. -
Intuitively, S (i. describes how much removing information from the region around location (i. changes the policy [23]. In other words, large (i.j) indicates that the information around pixel (i,j) important for the learning agent's decision making. Instead of calculating the score for every pixel, [33 found that computing saliency score for pixel i mod 5 and mod 5 produced good saliency maps lower computational cost for Atari games. The final saliency map P is normalized S.(ij)
as j)i,j) = Σ., S+(ta))
Comparison metries Next, we compile set of game images to compute human and RL saliency maps. For each game, we let trained PPO agent (with default hyperparameters) play the game until terminated, and uniformly sampled 100 images from the recorded trajectory. We will refer this set of images as the standard image set.
We then define Coefficient (CC) and Kullback-Leeble Divergence (KI.). Let Q denote the human saliency map predicted by the human attention network. CC is between -1 and 1 captures the linear relation
metrics for comparing saliency maps when using RL data: Pearson's Correlation
between two distributions Q and P:
CC(P, (PP,Q) Op X σQ
where (P,, Q) denotes the covariance, and OP and ØQ are the standard deviations of P and Q respectively. CC penalizes false positives and false negatives equally.
However, we may not want penalize the RI. agent if it attends regions that the human gaze model not currently focused on. The human gaze data only reveals the "overt" attention, and humans In other words, being attended to by the human gaze is a sufficient (but not necessary) condition for the features to be important. Thus we need second metric that penalizes the agent only if fails to pay attention to human attended regions, or equivalently, a metric that is sensitive to false negatives if we treat human attention as the ground truth. KL is an ideal candidate in this case 9
still pay "covert" attention to entities in the working memory [54 62
Q(t,5)
KL(P,Q) Q) EE
((, (e+
P(i,j)
where is small regularization constant (chosen to be 2.2204e-16 (9) and determines how much zero-valued predictions are penalized.
Results
To make meaningful comparisons, we first ensure that the human attention model is accurate, and that PPO agents' attention maps are consistent over repeated runs. We then compare human attention with PPO attention obtained from different learning stages and from agents that are trained with different discount factors. We then analyze PPO agents' attention in failure and unseen states. Finally, we show comparison results for other deep RL. algorithms.
Accuracy of human attention model We implement the convolution-dcconvoluu tion model 85 to generate human saliency map for image I at timestep t, given stack of four consecutive images I,-3, It-2, It-1 It as input. We use 80% gaze data for training and 20% for testing. The model can accurately predict the human gaze, On testing data, we obtained Area under ROC Curve (AUC) score of 0.968 0.005, CC of 0.562 + 0.030, and KL of 1.411 + 0.114 (n = 6), averaged over all games. Prediction accuracy for individual games can be found in Appendix of the supplementary materials, A visualization of the gaze data and prediction results can found in the attached video file. This accuracy is considered high in saliency research 9
gaze predic-

---

Consistency of RL attention We then show that saliency maps of the RL agents trained under the same experiment setting, provided by [30] are highly consistent despite the stochasticity in the training process. The stochasticity is controlled by random seed that is used to initialize both the game environment and the network. For each game, we use random seeds (0-4), train an agent using each seed, and generate 5 saliency maps with these trained agents. For each image in the standard set, we compute pair-wise CCs between the saliency maps (10 CCs total). The average value for these 10 CCs, across 100 images and 6 games, is 0924(=10000)
6000), Given such high consistency, the saliency maps we use later are the averaged results of these saliency maps.
4.1 RL versus human attention: The effects of learning
So far have verified that the gaze model can accurately predict human attention and that RL attention is consistent across repeated runs. We now address the primary research question: How similar are the visual features learned by RL agents and humans when performing the same task? We first study how the attention of PPO agents evolves over the training steps, compared with human attention. For each game, saved neural network weights different time steps during training. Then we use these saved models to generate saliency maps on the standard image set.
Cohalimes RD
8.9
0.8-
11.2
67-
Negutive
85-
--18
63
0.17
-31
10
32
120: 256
1924 1912
0 033 0.5
8.5
8.
0.59 E0500
Time Steps
Discount Factors
(a)
(b)
Figure 1: (a) Changes in human and RL attention similarity across learning time steps. PPO agents gradually learn to pay attention to important visual features and become more human-like, We also show comparisons between PPO agents and two control attention models: motion (optical flow) and saliency (low-level image features). The x-axis is log scale and KL values are negated for better visualization. The CC, KL, and score results for individual games, well as examples, can be found in Appendix 2. (b) Changes in human and RL attention similarity across different discount factors. Choosing different discount factors affects the RL agents attention and performance. The CC, KL, and score results for individual games, as well as more examples, can be found in Appendix
Image
Salieacy
O
(a) Freeway: The agent gradually learns focus attention on the yellow chicken crossing the highway.
CC-018LL-55 co-oweL-dds CD-OMAL-56t CC-40030-L31
CC-13482-2359
(b) Seaquest: The agent learns to attend to the yellow enemy on the right.
Attention of RL agents changes during learning and becomes more human-like. CC/KL values are caleulated between the RL agent's 's attention map and the human attention map.
Figure 2:
Fig. Ta shows the aggregated results for games. CC values and (negative) KL values between human and RL increase during learning, indicating that the RL agents' attention gradually becomes more human-like. We visualize the change RI. attention and human attention Fig. Networks without any training (time step 0) have saliency maps that positively correlated with humans
5

---

(CC= 0.170). For example, the second column in Fig. 2 shows that these networks are already sensitive low-level salient visual features without any learning, which is consistent with previous findings 79 23 Overall, human and RL saliency maps are more positively correlated after training (CC= 0.320: CC r(8) 0.992, p < 0.01; KI.: r(8) = 0.987. p < 0.01), meanwhile RI. attention is more similar to human attention than to the control models (higher average CC/-KL values after
training across all games).
We also show aggregated game performance in normalized game scores in Fig. aa For each game, we normalize the game scores obtained during learning (averaged over 50 episodes) by dividing them by the final scores. We find a strong positive correlation between the human CC/KL values and the game score (average Pearson's correlation coefficient of 0.813/0. 790).. The correlations for all games are statistically significant (p < 0.05, Appendix 2), indicating that small changes in similarity with human attention are reliable predictors of performance change. For comparison, the correlation values between CC/KL and game score for motion baseline are 0.404 20,2,, for saliency baseline are 0..58/ 0.688.
This result sheds light on an important research topic: bottom-up versus top-down attention. The two sides debate how much human or machine attention is driven by bottom-up image features captured by the saliency model [32], and how much it is driven by top-down task signals such as reward 660 8 Our result suggests that learning is key factor in this debate. In the early stages, attention is more driven by image features, indicated by the higher similarity between RL attention and saliency baseline. Then top-down reward signals shape the attention during learning by making reward-associated objects more salient and irrelevant objects less salient, as shown in Fig. 2
4.2 RL versus human attention: Discount factors
We then analyze how hyperparameters of the PPO algorithm affect the attention of the trained agents. We have seen that reward shapes attention during training, therefore a reasonable hypothesis that varying reward-related parameters will likely affect attention. One of the parameters we can vary in these games the discount factor ? E [0, 1). which determines how much the RL agent weighs future reward over immediate reward. The default is 0.99 for all games [30.. We train PPO agents with 7 E (0.1, 0.3, 0.5, 0.7, 0.9, 0.9999} and generate saliency maps on the standard image set. For each
? value, we normalize the game score by dividing it by the score obtained by the 7 0.99 agent.
Game Image
y 0.1 Cco 19
y=
==
y
7 0.9
Human
CC BL-LI
25
(a) Breakout: The human pays attention to the ball.
CONOUEE K1 K1-139
CO
CO
CCHE
CORMAILLL
CONMIRLLES COMMINLLL
(b) Seaquest: The human pays attention to approaching enemy from the left.
Figure 3: Effect of different discount factors on Ms.Pac-Man and Seaquest agents' attention. Agents trained with intermediate discount factor values have saliency maps that are more human-like.
The results are shown in Fig. lb Overall, RL attention is most similar to human's 's when 0.9. Beyond that, RL agents learn attend objects that matter only the long run. For example, Fig. Ba shows that in Breakout, RL attention is primarily on the ball (immediate effector) when Y 0.9. When 7 0.9999, however, it focuses on the number of lives, which does not indicate the end of the game until much later. With an intermediate value of 0.99, the agent attends to both groups of objects. On the contrary, humans rarely attend to long-term objects. This is because human attention is limited, so we have to keep those objects in the working memory and revisit them at lower frequencies (e.g., [7] 37). Therefore, these long-term objects do not show up our human attention model, and thus the less similar attention at larger discount factors.

---

Although in most games, the best game performance is achieved with default y 0.99, Fig. 16 shows that the best performance across games is obtained at 7 0.9. In Seaquest, the agent with 0.9 achieves 15% higher score than the default agent. Fig. 3b shows that with a lower the agent can focus, like humans, on an immediate threat from the left. Setting ? 0.99 or 0.9999 distracts the agent to attend to the oxygen bar at the bottom that is important in the long run but less urgent now. similar result was found for Pac-Man with an 18% higher score from the y = 0.9 agent than the 7 0.99 agent.
0.9.
This result suggests that deviating from the default y 0.99 can lead to better performance. Atari games are episodic tasks with true = 1, but lower values often lead to better performance in practice [63| 46 Fig. Iboss that Y 0.9999 agents perform poorly. We have verified that all 0.9999 agents (5 random seeds) converged after 200M samples, although to sub-optimal policies (Appendix 3). One reason for this deviation may be that being little myopic helps the agent focus on the most urgent targets. This result provides another reason, from a perception perspective, for why RL agents need adjust their planning horizon by reducing the discount contor-confirming the theoretical results provided by [36).
4.3 RL versus human attention: Failure states
We now turn to the second research question that concerns explainability in deep RL: Why do deep RL agents make mistakes? During training. RL. agents must learn to both identify relevant objects (perception) and make the correct decisions based on that information (policy). This leads to two types of states where RL agents could make mistakes: (1) they fail attend the right objects, and (2) they attend to the right objects but make the wrong decisions. Since human attention can be used identify important visual features, can mistakes? To answer this question, we compile failure image set by recording the game frames right before the RL agents lose a "life" which incurs a large penalty. We locate 100 such instances for each game. Freeway is excluded since the PPO agent learned policy that is nearly optimal.
human attention distinguish these two types of
Game Imaage
RL
Homan
Game Imange
RL
Humonn
CCHLIT
cc-aat
cC-oue kloeth
Montemana's Rec.
Mr. Par-Maa
-18
9.4-
trants Couse
8.3
A10
6.1-
Standard
Falluro
Standard
Falluro
Nagative
KL
Negative KL.
(b)
(d)
Figure 4: RL vs. human attention in states where RL agents made mistakes. (a) We show examples of RL and human saliency maps for Breakout (top: wrong attention; bottom: right attention but wrong decision), Montezuma's 's Revenge (wrong attention). and Ms.Pac-Man (right attention but wrong decision). (b) How attention similarities change failure states compared to normal states, ** indicates < 0.01. Error bars are the standard errors of the mean (n = 100). (c, d) Histograms of negative KL values for 100 standard vs. 100 failure states indicate that RL agents are more likely to neglect (Montezuma's 's Revenge) or attend to Pac-Man) human attended regions in failure states.
'Strictly speaking. 7 is a property of the MDP, not of the agent. Performance across MDPs with different rys are not directly comparable in this sense. Varying discount factor and clipping reward are common reward engineering designs that alter the true MDP return achieve better performance practice.
7

---

Fig. 4a illustrates the two types of failure states for Breakout (left column). In the first case, RL attention is extremely different from human's 's because it does not attend to the ball as the human model does. This lack of attention second case, the agent and the human model have highly similar attention maps because both attend to the ball. However, the agent fails to save the ball (which can still be saved), indicating that it has learned a suboptimal policy despite good perception. Following this method, we can use human attention as reference to quantitatively interpret RL failure cases: if RL attention is in general less similar to human attention failure than in normal states, then the model less frequently attends to task-relevant objects ("bad perception") "); if it is more similar, then the model is not making the correct decisions despite the correct visual information ("bad policy").
task-relevant objects likely contributes to its failure. In the
Fig. 4b shows the quantitative results for all games. For Montezuma's 's Revenge, the RL attention in failure states becomes less similar to human attention compared to the standard image set, indicated by significantly decreased CC/KL, values. Fig. Pig.[c] shows the (negative) KI. histogram of 100 failure states vs. normal states. We can see that the failure states the RL. agents are more likely to neglect human attended regions. Fig. 4a shows that the agent fails to attend to the enemy as the human does. With the same amount of training, the Montezuma agents do not learn to identify task-relevant objects in the game. In other words, they do not learn to parse out the semantics and structures of the game. This suggests that Montezuma's 's Revenge is perceptually more difficult for the agents.
On the contrary, in the failure states for Ms. Pac-Man and Seaquest (Fig. 4bj, attention maps of RL agents and humans are more similar than in the normal states. Fig. 4d shows the (negative) KL histogram of 100 failure states vs. normal states from Ms. Pac-Man, in which humans and RI agents are more likely to agree on the objects to attend to failure states. An example frame is shown on the bottom of Fig- 4al the agent attends to the Pac-Man and the enemy ghost similarly to humans, but it makes the wrong decision and runs into the ghost. Compared with Montezuma' 's revenge, Ms. Pac-Man is perceptually easier for the agents.
We conclude that similarity measurements with human attention may help identify and interpret the failures made by RL agents, as well as identifying tasks that are more perceptually difficult to learn for the agents.
4.4 RL versus human attention: Unseen data
Next, we study whether the RL agent's attention generalizes to unseen states. The unseen states late-game states obtained from human experts' data 855 which RL agents have not encountered (i.e., above the agents' best score; score threshold for each game be found in Appendix 5). We refer to this set as the unseen image set (100 images per game). Again, Freeway is excluded since the agent nearly optimal.
Gauee Image
RL
Humau
Mastzzann'ss
Seuquea Monterma
XL-3.13
-1.
Pearun's
R...
CC-057 KL-001
Staedard Useen
Staedard Usscee
(a)
(b)
Figure 5: Human versus RL attention in states that RL agents have not seen. (a) How attention similarities change in unseen states compared to seen states in different games. (b) Unseen states for Montezuma'ss 's Revenge (top), the RL agent's 's attention is very different from human attention due to new object on the left. For Seaquest (bottom), they are similar.
Fig. 5a shows the results. For Frostbite and Montezuma's 's Revenge, the similarities drop significantly. One reason for this drop may be the presence of new objects that the agents have never encountered in the unseen states. Fig. Fig-5blshi 5b shows an example for Montezuma's Revenge. The agents attended to the ladder, a previously seen object, but failed to attend to a new enemy object on the left like the human did. For Seaquest and Ms. Pac-Man, on the other hand, RL attention and human attention
8

---

more similar on unseen data. Compared with the game structure in Montezuma's 's Revenge, Seaquest and Ms. Pac-Man do not have new objects in the unseen states- the objects only move faster and appear in larger numbers. However, this alone only supports lack of decrease in attention similarity. We suspect the increase in similarity may be due to dangerous states similar to those in Section but more controlled experiments (e.g., suggested in (3) are required to further interpret this result. Breakout is an interesting case. The CC value drops significantly in unseen states whereas the negative KL value does not change much. The similar KL values may be attributed to the maintained attention on the ball and paddle in unseen states, whereas the decreasing CC values may be due to unseen spatial layouts of the bricks. Again, more controlled studies required to confirm this. More examples are in Appendix 5.
The results suggest that one of the first obstacles to achieving expert human-level performance for certain games is the perception challenge- the agents need to learn to recognize, attend to, and then learn to act upon new objects. This is easy for humans due to their prior knowledge but challenging for RL. agents 440 74 18. For the other games without novel objects, the agent's attention is fairly generalizable and it needs to learn good policy for challenging states.
4.5 Other Atari agents
The above analyses were done for a particular RL. aPoorithm-PPO. Next we apply our method to
other RL algorithms, including C51 (5). Rainbow (28), DQN [46. and A2C [45], as well as two
evolutionary algorithms, GA [69 and ES [1]. We use trained models from Dopamine [[]] and Atari Model Zoo 70) in which each algorithm has 3-5 trained models. There is strong positive correlation between model performance (in terms of the game score, averaged over 50 episodes each) and similarity measurement (in terms of CC with human attention on the standard image set). The average correlation coefficient for five games is r = 0.631 (excluding Montezuma since most algorithms have zero scores). The results for individual games can be found in Appendix 6. This result suggests that the overall positive correlation between model performance and similarity with human attention generalizes other deep RL algorithms. Although not performed in this work due to resource limitations, future studies should run experiments on the effect of learning and discount factors to further confirm our findings.
5 Discussion
We provide visual explanations for deep RL agents using human experts' attention as reference. We have discussed how RL attention develops and becomes more human-like during training, and how varying the discount factor affects learned attention. We show that human attention can be useful in diagnosing an RL agent's 'ss failures. We also identify challenges in closing the performance gap between human experts and RL agents in the unseen states experiment. Our analysis is restricted to saliency map comparisons, but other approaches are possible for measuring the similarity of representations learned by different RL agents [77]. Our human attention models, all compiled datasets, and tools for comparing RL attention with human attention are made available for future research in this direction.
When interpreting the results, it is important to keep in mind the limitations of saliency- based explainable RL methods [7T] M. Recent studies have shown that saliency maps are more exploratory rather than explanatory 3 To infer causal explanations, one can start with using saliency maps to build falsifiable hypotheses, and perform counterfactual evaluations of these hypotheses, e.g., by manipulating states to generate counterfactual semantic conditions 3 Our framework helps direct researchers' efforts towards data that deviate from human attention patterns, from which useful hypotheses more likely to arise.
For researchers who are interested in RL algorithms, we have gained least three important insights. First, since the task performance and similarity to human attention are highly correlated, one could use human attention as prior knowledge to guide the learning process of RL agents, e.g., by encouraging them to attend to the correct objects early in the learning process. This could be especially helpful for games like Seaquest, in which the agent has not learned to focus on the right object after 5120k time steps (Fig. 2). Two human studies using Atari games suggested that prior knowledge, such as perceptual prior, is why humans learn faster and better in these games 18). Then the next question is how to incorporate human attention into which has been studied in several computer vision

---

tasks 157]. Our results indicate that humans typically attend to fewer regions than RL agents do. Therefore desirable loss function should encourage the agents to focus on the regions that humans attend to, but would not penalize the agents for attending to more regions [62
Second, our results provide visual explanations for the agents' performance when varying the discount factors and highlight the importance of choosing proper planning horizons with appropriate discount factors. Recent works confirm this by showing that beneficial have an adaptive discount factor 4 or multiple discount factors 21
Third, failure analysis could identify tasks and states where RL agent's 's attention drastically differs RL paradigm, these states may need human intervention
from expert humans. In a camaa--n--the-aa correction. By using method for comparing human and RL attention, researchers may diagnose their algorithms with the states of their interest.
For researchers who are interested in using RL as models for cognition, it is perhaps both surprising and encouraging to see that RL agents trained from scratch with only images and reward signals can develop attention maps that are similar to humans, especially when considering that they have very little prior knowledge. This result similar to previous research that shows CNNs trained from image classification tasks can learn features that are similar to the ones in the visual cortex (19, 78, 79]. Consistent with previous findings, we show that model task performance and feature similarity are highly correlated [79]. Our results are complementary the recent findings using human brain imaging data when playing Atari games [3), suggesting that deep RLs can learn biologically plausible representations and can be used models for human gaze, decision, and brain activities.
Multiple factors are important for interpreting our results and could explain the remaining differences between human attention and RL attention. The first one regards the nature of human attention. Humans store information in memory and do not need to constantly move their eyes to attend to all task-relevant objects. To complete the human attention map, in addition to gaze data (overt attention), one will need to retrieve human covert attention from brain activity data, a technique that became possible recently 13. Another factor is the human intrinsic reward. Humans are likely to have internal reward functions that are different from the ones provided by the game environment, and reward is known affect attention [60] 41). Hypothetically specific algorithmic network architecture designs that capture important cognitive aspects of human decision-making could lead to more similar saliency maps.
closely related research direction compares a human player's policy with an RL agent's learned policy 47 which could further allow us to better understand the similarities and differences between humans and RL agents. However, as we have shown here, the difference in decisions could be due to perception, which needs to be considered while comparing policies. Our approach lays the groundwork for future research in this direction.
Ethics statement Our work studies how attentional mechanisms of humans and decision-making reinforcement learning agents are similar or different. The hope is that our work will help us better understand the differences and similarities between humans and AI. Comparing humans with Als allows us to better understand their strengths and limitations. The study involves collecting and modeling human eye-tracking data. We have removed all personally identifiable information such that the private information of human subjects is protected.
Acknowledgments and Disclosure of Funding
This work is supported by NIH Grant EY05729; A portion of this work has taken place in the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (CPS-1739964, 1724157,, FAIN-2019844), ONR (N00014-18 1000014-18-2221 8--2243), ARO (W91 19--19-3933-), DARPA, Lockheed Martin, GM, Bosch, and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
10

---

References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 9525-9536, 2018
[2]
Alnour Alharin, Thanh-Nam Doan, and Mina Sartipi. Reinforcement learning interpretation methods: A survey. IEFE Access, 8:171058-171077 2020.
[3] Akanksha Atrey, Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis saliency maps for deep reinforcement learning. International Conference on Learning Representations, 2019.
[4] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvit- skyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[5] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, pages 449-458 PMLR, 2017.
[6] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[7] Ali Borji, Ming-Ming Cheng, Huaizu Jiang, and Jia Li. Salient object detection: A benchmark. IEEE Transactions on Image Processing. 24(12):5706-572) 2015.
[8] Ali Borji and Laurent Itti. Defending yarbus: Eye movements reveal observers' task. Journal of Vision, 14(3):29-29, 2014.
(9] Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand. What do different evaluation metrics tell about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence, 4133):740-757, 2019.
[10] Zoya Bylinskii, Adrià Recasens, Ali Borji, Aude Oliva, Antonio Torralba, and Frédo Durand. Where should saliency models look next? In European Conference on Computer Vision (ECCV), pages 809-824. Springer, 2016.
[11] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Belle- mare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018.
[12] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vinceth N Balasubramanian. Grad-cam++: Generalized gradient- based visual explanations for deep convolutional networks. In 2018 JEEE Winter Conference on Applications of Computer Vision (WACV), pages 839-844 -47. IEEE, 2018
[13] Logan Cross, Jeff Cockburn, Yisong Yue, and John O' Doherty. Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments. Neuron, 2020.
[14] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pages 6967-6976. 2017.
[15] Abhishek Das, Harsh Agrawal, Larry Zitncck, Devi Parikh, and Dhruv Batra. Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding, 163:90-100, 2017.
[16) Tao Deng, Hongmei Yan, Long Qin, Thuyen Ngo, and BS Manjunath. How do drivers allocate their potential attention? driving fixation prediction via convolutional neural networks. IEEE Transactions Intelligent Transportation Systems, 2019.
[17] Gabriel Diaz, Joseph Cooper, Constantin Rothkopf, and Mary Hayhoe. Saccades to future ball location reveal memory-based prediction in a virtual-reality interception task. Journal of Vision, 13(1):20-28 20, 2013.
11

---

[18] Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L Griffiths, and Alexei A Efros. Investigating human priors for playing video games. arXiv preprint arXiv:1802. 2018.
[19] Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage. 152:184-18 18-11,,, 2017.
[20] Gunnar Farnebäck. analysis, pages 363-370, 2003.
motion estimation based on polynomial expansion. Image
[21] William Fedus, Carles Gelada, Yoshua Bengio, Mare G Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons, arXiv preprint 1XX2::992.00880 2019.
[22] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pages 3999- 2017.
[23] Samuel Greydanus. Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and under- standing atari agents. In International Conference on Machine Learning, pages 1 1792-1801, 2018.
[24] Christopher Grimm, Dilip Arumugam, Siddharth Karamcheti, David Abel, Lawson LS Wong, and Michael I. Littman. Modeling latent attention within neural networks. arXiv preprint 2017.
arXiv:
[25] Piyush Gupta, Nikaash Puri, Sukriti Verma, Sameer Singh, Dhruv Kayastha, Shripad Deshmukh, and Balaji Krishnamurthy. Explain your move: Understanding agent actions using focused feature saliency. arXiv preprint arXiv: 1912. 12/91, 2019.
[26] Sen He, Hamed R Tavakoli, Ali Borji, Yang Mi, and Nicolas Pugeault. Understanding and visualizing deep visual saliency models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10206-10215, 2019.
[27] Sen He, Hamed Tavakoli, Ali Borji, and Nicolas Pugeault. Human attention in image captioning: Dataset and analysis. In Proceedings of the IEEE International Conference on Computer Vision, pages 8529 8538, 2019.
[28] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Shirty-Second AAAI Conference Artificial Intelligence, 2018
[29] Alexandre Heuillet, Fabien Couthouis, and Natalia Dfaz-Rodríguez Explainability in deep reinforcement learning. Knowledge-aased Systems, page 106685, 2020.
[30] Ashley Hill, Antonin Raffin, Maximilian Emestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert. Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https 7 //github. www.hilloooheceeccccccccccececca aallaaa/ssabeeee wttpsstaabbbbbeeeeeeeeeeeeeeeeee 2018.
[31] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532(7600):433-558, 2016.
[32] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1244 1998.
[33] Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Trans- parency and explanation in deep reinforcement learning neural networks. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 144-150, 2018
12

---

[34] Shailee Jain and Alexander Huth. Incorporating context into language encoding models for finri. In Advances in Neural Information Processing Systems, pages 6628 6637, 2018.
[35] Theo Jaunet, Romain Vuillemot, and Christian Wolf. Driviz: Understanding decisions and memory in deep reinforcement learning. arXiv preprint
2019.
[36] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1181-1189, 2015.
[37] Leif Johnson, Brian Sullivan, Mary Hayhoe, and Dana Ballard. Predicting human visuomotor behaviour in driving task. Philosophical Transactions of the Royal Society B: Biological Sciences,
2014.
[38] Ho-Taek Joo and Kyung-Joong Kim. Visualization of deep reinforcement learning using grad- cam: How ai plays atari games? In 2019 IEEE Conference Games (CoG), pages 1-2. IEEE, 2019.
[39] Qiuxia Lai, Wenguan Wang, Salman Khan, Jianbing Shen, Hanqiu Sun, and Ling Shao. Human ttxtit {vs) machine attention in neural networks: A comparative study, arXiv preprint 1906::9906.08644 2019.
[40] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
[41] Yuan Chang Leong, Angela Radulescu, Reka Daniel, Vivian DeWoskin, and Yael Niv. Dynamic interaction between reinforcement learning and attention in multidimensional environments. Neuron, 93(2):451 463,,4663222 2017.
[42] Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In European Conference on Computer Vision (ECCV), pages 619-635, 2018.
[43] Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-255 2016.
[44] Josh Merel, Diego Aldarondo. Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Ölveczky. Deep neuroethology of virtual rodent. arXiv preprint arXiv:1911.09455 2019.
[45] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lilli- crap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937, 2016.
[46] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):52--53. 2015.
[47] Steven Moore and John C Stamper. Exploring expertise through visualizing agent policies and human strategies in open-ended games. In EDM ( (Workshops), pages 30-37, 2019.
[48] Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Advances in Neural Information Processing Systems, pages 12329-12338, 2019.
[49] Sajad Mousavi, Michael Schukat, Enda Howley, Ali Borji, and Nasser Mozayani. Learning to predict where to look in interactive environments using deep recurrent q-learning. arXiv preprint arXiv: 2016.
[50] Satya M Muddamsetty, Mohammad NS Jahromi, Andreea E Ciontos, Laura M Fenoy, and Thomas B Moeslund. Introducing and assessing the explainable ai (xai) method: Sidu. arXiv preprint 2rXi 2021.
[51] Tam Nguyen, Qi Zhao, and Shuicheng Yan. Attentive systems: A survey. International Journal of Computer Vision, 126 2018
13

---

[52] Dmitry Nikulin, Anastasia lanina, Vladimir Aliev, and Sergey Nikolenko. Free-lunch salieney via attention in atari agents, arXiv preprint arXiv:1908.025:: 020010 2019.
[53] Andrea Palazzi, Davide Abati, Simone Calderara, Francesco Solera, and Rita Cucchiara. Pre- dicting the driver's focus of attention: the dr (eye) project. IEEE Transactions on Putlern Analysis and Machine Intelligence, 2018.
[54] Michael I Posner. Orienting of attention. Quarterly Journal of Experimental Psychology. 3211) 1980.
[55] Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: A survey. In Inter- national Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 77-95. Springer, 2020.
[56] Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krish- namurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations, 2019.
[57] LAI Qiuxia, Salman Khan, Yongwei Nie, Sun Hanqiu, Jianbing Shen, and Ling Shao. Under- standing more about human and machine attention in deep neural networks. IEEE Transactions on Multimedia, 2020.
[58] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1135-1144, 2016.
[59] DA Robinson. The mechanics of human saccadic eye movement. The Journal of physiology. 174(2):245-264, 1964.
[60] Constantin A Rothkopf, Dana H Ballard, and Mary M Hayhoe. Task and context determine where you look. Journal of Vision, 7(14):16-16, 2007.
[61] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint
038644 2017.
[62] Akanksha Saran, Ruohan Zhang, Elaine Schaertl Short, and Scott Niekum. Efficiently guiding imitation learning algorithms with human gaze. arXiv preprint arXiv:2002. 2020.
[63] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
[64] Ramprasaath R Selvaraju. Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626. 2017.
[65] Wenjie Shi, Zhuoyuan Wang, Shiji Song. and Gao Huang. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. arXiv preprint arXiv:2003.07065 2020.
[66] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning- Maaning-Vollme 70, pages
JMLR. org, 2017
[67] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034. 2013.
[68] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: #rXiv:/4/2.6806, 2014.
[69] Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning, arXiv preprint
2017.
14

---

[70] Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Jiale Zhi, Ludwig Schubert, Mare G Bellemare. Jeff Clune, et al. An atari model zoo for analyzing. visualizing, and comparing deep reinforcement learning agents. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3260-3267. AAAI Press, 2019.
[71] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Bearning-Volumee 70, pages 3319-3328. JMLR. org, 2017.
[72] Hamed R Tavakoli, Rakshith Shetry, Ali Borji, and Jorma Laaksonen. Paying attention to descriptions generated by image captioning models. In Proceedings of the IEEE International Conference on Computer Vision, pages 2487-2496, 2017.
[73] Jan Theeuwes, Artem Belopolsky, and Christian NL Olivers. Interactions between working memory, attention and eye movements. Acla psychologica, 1 132(():106-114, 2009.
[74] Pedro Tsividis, Thomas Pouncy, Jacqueline Xu, Joshua B Tenenbaum, and Samuel Gershman. Human learning in atari. 2017.
[75] Junpeng Wang, Liang Gou, Han-Wei Shen, and Hao Yang. Dqnviz: visual analytics approach to understand deep g-networks. IEEE Transactions on Visualization and Computer Graphics, 25(1):288-298, 2018.
[76] Laurens Weitkamp, Elise van der Pol, and Zeynep Akata. Visual rationalizations in deep reinforcement learning for atari games. In Benelux Conference on Artificial Intelligence, pages 151-165. Springer, 2018.
[77] Erik Wijmans, Julian Straub, Dhruv Batra, Irfan Essa, Judy Hoffiman, and Ari Morcos. Ana- lyzing visual representations embodied navigation tasks. arXiv preprint arXiv:2003.05993 2020.
[78] Daniel L Yamins, Ha Hong, Charles Cadieu, and James J DiCarlo. Hierarchical modular optimization of convolutional networks achieves representations similar to macaque it and human ventral stream. In Advances in Neural Information Processing Systems, pages 3093- 3101, 2013.
[79] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. cetformance-optimiize hierarchical models predict neural responses higher visual cortex. Proceedings of the National Academy of Sciences, 1 111(23):8619 8624, 2014.
[80] Zhao Yang, Song Bai, Li Zhang, and Philip HS Torr. Learn to interpret atari agents. arXiv preprint arXiv: 1812. 1276, 2018
[81] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pages 318-833. Springer, 2014.
[82] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 26(110):084-1100) 2018.
[83] Ruohan Zhang, Zhuode Liu, Luxin Zhang, Jake A Whritner, Karl S Muller, Mary M Hayhoe, and Dana Ballard. Agil: Learning attention from human for visuomotor tasks. European Conference Computer Vision (ECCV), pages 692-707. Springer, 2018.
[84] Ruohan Zhang, Akanksha Saran, Bo Liu, Yifeng Zhu, Sihang Guo, Scott Niekum, Dana Ballard, and Mary Hayhoe. Human gaze assisted artificial intelligence: A review. In IJCAI: proceedings of the conference, volume 2020, page 4951. NIH Public Access, 2020.
[85] Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl S Muller, Jake Whritner, Luxin Zhang, Mary M Hayhoe, and Dana H Ballard. Atari-head: Atari human eye-tracking and demonstration dataset. In Thirty-Fourth AAAI Conference on Artificial Intelligence. AAAI Press, 2020.
15

---

[86] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2921-2929, 2016.
[87] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] Please Sec.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see Ethics Statement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] The paper does not include theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] The paper does not include theoretical results
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] Please Sec. and Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Sec. B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Please see ethics statement.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] The paper does not involve human subjects.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] The paper does not involve human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] The paper does not involve human subjects.
16


Reviews:
[{"review_id": "wOghKyvbUul", "reviewer": "Reviewer_sG7Q", "summary": "The paper compares RL agent attention with human attention on various Atari games. Correlations between RL agent and human attention with learning performance, failure states, discounting, and generalization are examined.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Coincidentally two years ago, I wanted to explore the same research question as the paper: \"How similar are the visual features learned by RL agents and humans when performing the same task?\" In fact the primary domain I was interested in experimenting on was the Arcade Learning Environment, presumably the same set of Atari-based environments used by the paper. However a key obstacle was the lack of relevant human attention data on Atari environments. The paper attempts to address this by using what they denote as \"human expert gaze data\" from the Atari-HEAD dataset. This falls short for reasons explained in the section labeled \"Quality\" directly below. \n\nThe paper claims that they \"present the first study that compares the RL agent's attention with human attention.\" The paper does present some interesting results on correlations and KL divergences between changes in human and RL attention over training. However the results on failure states and unseen data don't have clear trends, and the papers provides only speculations as to why many of the results are what they are.\n\nQuality: The paper is fundamentally based on comparisons between RL agent attention and human attention, which is inferred from \"human expert gaze data\" from the Atari-HEAD dataset. Regarding this dataset, the paper states \"The original game runs continually at 60Hz [4], a speed that is challenging even for professional gamers. Human eye movements were rushed and inaccurate at this speed hence could not serve as a useful reference. In [81], however, the human data were collected in a semi-frame-by-frame mode, a design that allowed enough time for the players to attend to all relevant objects on the screen and make decisions. The slowed speed allowed players to achieve world expert level performance [81] which is about 35x better than the human performance reported in previous deep RL literature [43, 26].\" Therefore this isn't a comparison involving human attention in a realistic setting but instead a significantly altered setting that could result in markedly different human attention. Though some experiments are interesting, and some show trends in the results, they generally lack robustness: even for the experiments with trending results, it's unclear how the user can benefit from such findings.\n\nClarity: The paper is well written and easy to understand.\n\nSignificance: This has been covered in the sections labeled \"Originality\" and \"Quality\".\n\nI believe this paper has good intentions with the work it puts forth, but I believe that the experiments could be made stronger in order to find potentially actionable insights. Furthermore, I believe the paper could dive deeper into its results in order to put forth claims that are supported by resulting evidence instead of putting forth speculations. Maybe most importantly, the paper should consider using human attention data that is representative of realistic settings instead of significantly altered ones unless there are guarantees that human attention remains the same between the two.\n\n=====\n\n[9/1/2021] Update: My response to the Authors' rebuttals can be found below. I'm increasing my rating from a 4 to a 5.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the paper acknowledges that many of its claims are speculative and provides an \"Ethics statement\" explaining their handling of sensitive data.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "vyIrwErZxtO", "reviewer": "Reviewer_HwvK", "summary": "The authors present a comparison of a model trained on human gaze data when playing Atari games and a PPO agent in terms of saliency maps. The authors run a large number of experiments and identify a number of interesting results in terms of where the saliency maps match and do not match.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: As the authors state, they are the first to undertake research like this, to the best of my knowledge. The results are also novel, though many reaffirm findings from elsewhere. Overall, the work is of sufficient originality for inclusion. \n\nQuality: The work is of very high quality. The authors present deep, in-depth experiments and rigorous analysis. The appendices are particularly helpful here and would be especially pertinent for future researchers. My only concern when it comes to quality is the relative lack of information on the human gaze model. While this is understandable given space limitations, there's a few details I'd like more detail on. First, how was data split? The authors state they used an 80-20 split, but with no further details. If it was by a random sampling I would have concerns, as it's possible the same human individuals ended up in both the training and test datasets, which could skew the metrics in favour of the model. Second, the authors are quite vague on the model and the training process for it, stating only \"we follow their approach to train models with human gaze data\". Finally, I'd appreciate more detail on the dataset in question.  \n\nClarity: I have a couple of concerns when it comes to the clarity of the work. First, the authors do not make clear until page 3 that they aren't directly comparing RL saliency maps and human gaze data, While in hindsight I can identify why this would be challenging, the authors never make the case why this is necessary. Throughout the paper the authors refer to their work as comparing humans and RL agents, instead of a proxy for humans. There are also some oddly worded sentences that I can't quite parse. For example \"By publishing our human gaze prediction model researchers can diagnose their algorithms with the states of their interest.\". \n\nSignificance: I think any RL researcher would be interested in this work, with particular interest coming for researchers who employ the Atari domain. The findings are novel and valuable with several promising directions for future work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "All major limitations of the paper are addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "IbxdZz7zqMn", "reviewer": "Reviewer_bEDj", "summary": "The authors investigate the (dis)similarity of the saliency maps of reinforcement learning agents versus visual attention maps of humans performing the tasks of learning to play Atari games for PPO, as well as confirming that the conclusion of positive correlation between the model performance and similarity for other algorithms. The paper poses (and gives some evidence on) the question: do deep RL agents pay attention to the same visual features as humans do?\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Related work:*\n\nThe related work covers different aspects of previous work and acknowledge dual aspect of this work, namely explainable reinforcement learning and studies of human attention. \n\n*Originality and significance*\n\nWhile the work compares (mostly) the existing PPO method using existing saliency maps for interpretable reinforcement learning (RL), it gives a valuable insight about the relation between human and RL attention for PPO methods.  The work lies at the intersection of interpretable RL and human attention, and it would help build upon current understanding of failure modes of the RL methods such as PPO. It could be also extended to the other methods to see whether the same results are maintained for different RL algorithms. \n\n*Pros:*\n\n The paper studies the perceptional difference between estimated human attention and RL agents  and links it to the failures of control of the RL agents. This is an important problem, and the authors shed the light on it using a well-defined methodology\nThe paper is clearly written (although some outstanding questions remain) and gives a comprehensive overview of the state-of-the-art methods. The paper demonstrates the relation between human and PPO model attention in a case of different hyperparameters and  tasks; it also sheds a light on what happens during the failure state. \n\n*Cons:* \nThe experiments are carried out mostly for the PPO algorithm (except for correlation analysis in Appendix 6 which only shows that the policy improvement leads to better correlation of human vs RL attention); it would be interesting to know more about how it generalises to other methods but I accept the authors explicitly propose it to be a part of the wider study and the current work has enough merit, in a view of the reviewer\n\n*Other comments:*\n[1] \"In order to capture variance in training and ensure reproducibility, we select six popular 119 Atari games instead of using all Atari games and train 630 agents in total (300 GPU days on GeForce 120 GTX 1080/1080 TI).” How many agents per each game, is it 105? Are settings the same?\n\n[2] “The unseen states are late-game states obtained from human experts’ data [81] which RL agents have not encountered.” The reviewer thinks would be good to clarify what the word late would mean in this context, i.e. how exactly these late-game states have been selected, was it some sort of threshold on the frame number or something else? \n\n[3]One of the potential limitations for the most of the experiments is that they compare RL and *predicted* human attention. It has been addressed in the appendix 7 by extracting human fixations and it shows that the fact that the conclusions stay the same (from Appendix 7: *In general, the agents’ attention becomes more similar to humans’ as training progresses* ) for the actual gaze  collected using the EyeLink post-processing software to extract the fixations. The reviewer is wondering on whether it was collected on the whole data[81] or what was the subset of data? Also, it was the reviewer's impression from the main text that the dataset contained gaze fixations and you performed the attention map prediction using convolutional-deconvolutional network based on these fixations, so why was EyeLink software needed for the experiment in Appendix 7 but not for the main text? \n\n[4] \"Fig. 4c shows the (negative) KL histogram of 100 failure states vs. normal states in which the human and RL agents are more likely to agree on the objects to be attended to.” It sounds like an interesting case. In the appendix, it is shown that while the attention map makes sense, the failures are caused by the policy. Would it be possible to clarify whether it was caused by the stochasticity of the policies or do the authors see any other explanation?\n\n[5]  Fig. 1: would it be possible to reference to the exact methods used for control attention models (optical flow and saliency) as the reviewer would struggle to reproduce this experiment just from the text?\n\n**Update**\n\nScore increased to 7 as a result of rebuttal discussions as the authors cleared a number of concerns (see below). \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper analyses mostly the performance of PPO (apart from Appendix 6 showing the correlation plots for other methods); the authors acknowledge these limitations and are fair in their assessment of the result. The ethics judgement also seems reasonable to the reviewer, and the data used in the paper appear to be anonymised. ", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "-dlFJi6mubQ", "reviewer": "Reviewer_hVFS", "summary": "This work proposes to use human gaze data and an attention model derived from it to compare and contrast human attention and attention saliency maps extracted from PPO deep RL agents on Atari Learning Environment games. The authors analyze the effect of learning, discovering that learning improves similarity between agent attention and human attention. The authors evaluate how the discount factor contributes to similarity, finding that attentional similarity and performance peak simultaneously, with fairly high discount factors. The authors also perform some focused analyses of how attention varies in failure states for these artificial agents, and how attention behaves when attempting to generalize to unseen data from the same games. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Post-discussion notes:** Following the authors' responses and discussion with the other reviewers, I'm revising my rating to 7. I still believe there's room to ablate some of the decisions on how you constructed the metric, but I believe it captures something meaningful as-is, alleviating my primary concern with the paper.\n\n**Originality:** I find the approach interesting and original, clearly building on the gaze-related work of [81], and expanding to perform meaningful success and failure analyses of PPO deep RL agents. This work makes intriguing use of the human data and model derived from it.\n\n**Quality:** While the work seems to have been approached with rigor in mind, I have several questions regarding some decisions made, that impact my confidence in the rigor of the paper as a whole. One issue is the choice of loss on the saliency maps — given that the entire set of qualitative analyses relies on the saliency metric, additional clarity on different choices made with the saliency metric would be quite helpful [see additional details below]. In addition, several of the claims made in sections 4.3-4.5 appear to be generalizations from a single observation (or if they are from visually examining a larger set of examples, it is not made clear). Providing metrics to quantify some of these effects (4.2, 4.4) would go a long way in providing rigor. Finally, while I understand the technical limitations underlying the decisions made in section 4.5, providing evidence that one could replicate this effort to additional model architectures is different from providing evidence that the results of these analyses are likely to generalize to other models.  \n\n**Clarity:** While the overall structure of the paper is sensible, I found the paper unclear in several locations. See more detailed notes and questions below. The term ‘feature’ is used for distinct things in Sections 1 and 2, making it harder to navigate. Some detail on the models of human attention (exact inputs and outputs) is lacking. The organization of some of the results sections is odd (see comments below), and it is hard to appreciate if some conclusions drawn were the result of a single visualized image, a visual inspection, or what sort of process. \n\n**Significance:** The work provides interesting and novel data on how attention saliency maps produced by a particular deep RL algorithm align with attention maps produced by a model of human attention on the same data, including analyses of the effect of the course of learning, the discount factor, and several tailored analyses of failure states and generalization. These methods may prove interesting for others developing methods and evaluating them on ALE games, at least for the games for which the authors provide a trained human attention model. Building on this work for other domains or tasks would provide substantially harder, as it would require collecting eye-tracking data for the specific domain (and training an attention model on it).  \n\nI look forward to reading the authors' responses and additional reviews. \n\n### Detailed Notes\n\n* **Sections 1/2:**\n    * The term ‘feature’ is used often and appears to refer to different aspects of the problem. The abstract and introduction discuss the features learned by humans and machines, that is, some representation or transformation of the visual input. Later in the abstract, the authors refer to ‘visual features’ of Atari games, and in the related work section, discuss the human ability to to ‘move their foveae to the right place at the right time to attend to important features,’ referring to aspects of the input. I would suggest the authors pick different terms to refer to each of these distinct things (as one is a learned representation of the other), as that would clarify some of the language early in the manuscript.\n    * Line 77: ‘alternating the input’ — should be altering?\n* **Section 3:**\n    * Lines 89-95: the authors mention that the human gaze data used was collected (in [81]) in a semi-frame-by-frame mode, which allows human participants to process the input for as long as they’d like before selecting an action. Given the ability to process each frame for as long as they desire, is the comparison between agents and humans in this case (even for the purposes of extracting attention) valid? Doesn’t this imply that the agents and humans are solving somewhat different problems?\n    * Lines 102-112: models of human attention — the authors could consider clarifying what the output of these models are — do they produce a saliency map from the gaze data and input frame? If not, what do they produce? Understanding this detail should not require jumping through the references. \n    * Line 125: the choice of a loss for the saliency map — is there a particular reason why the L2 loss is chosen? Assuming the policy is normalized to a valid distribution, might a divergence metric (KL, etc.) make more sense? Two follow-up questions about the choice of metric:\n        * Not all changes to the policy are equally distinct. Imagine the case where the pre-perturbation policy is 100% move left. If the post-perturbation policy is 100% move right, that makes for a substantially larger change than if the post-perturbation policy is 100% move up and left simultaneously. However, these two changes entail the same L2 distance. Perhaps the metric should take into account the functional effect of the actions, and not merely the change in distribution?\n        * Another issue, in some ALE games, comes with the FIRE action, which can be coupled with any other action (see the action space defined in constants here: https://github.com/openai/atari-py/blob/master/atari_py/ale_interface/src/common/Constants.h). While many games declare the full action space, if firing is meaningless in a game, it could be the case that there exist actions that are functionally identical (for example, PLAYER_A_UP and PLAYER_A_UPFIRE) but are represented by different elements in the policy vector, such that shifting mass from one to another retains the same functional policy. Is this an issue with any of the games evaluated? \n    * Lines 139-147: given the fact the humans can attend to as much of the image as they would like to, while the agent is limited to a single pass, was the opposite KL divergence (KL QP) considered, that would only penalize the agent if it attended to regions the humans did not attend to? \n    * This might require combining all fixations/gaze samples for each player and input image to a single input, which does not sound unreasonable, but I am not aware whether or not [81] performed such preprocessing.\n* **Section 4:**\n    * Lines 155-156: perhaps I missed this detail in other sections of the paper, but — given the standard approach of stacking four consecutive frames, but the human having access to a single frame at a time to view for as long as they desire (and hence, their gaze is presumably less influenced by previous frames), why is it sensible to compute the human saliency maps from a set of four consecutive images? \n    * Lines 162-169: the consistency of the saliency maps across random seeds seems a little surprising. Does this occur often with saliency maps in computer vision or similar approaches in language processing, or do those show greater variability across random seeds?\n* **Section 4.1/Figure 1:**\n    * The text should be much larger. If the manuscript were to be printed out, it would be near-impossible to read. This comment stands for all figures. \n    * Is the comparison to motion and saliency (the green and red lines) between the RL agents and these alternative methods or the human data and these alternative methods? I assume it is the former, but perhaps the caption should clarify.\n    * Is there a hypothesis for why the human comparisons show a minimal gap between the CC and the KL, but the other two methods show a substantial gap between the two evaluation metrics?\n    * Lines 186-193: I am not sure I understand the statement regarding the correlation values between CC/KL and game scores for the baselines. Examining the data in appendix 2, figure 1, for Breakout, the correlations between game score and motion, and game score and saliency, are statistically significant when using cross-correlation. Could this statement be clarified? \n* In Figures 3-5, when human attention is plotted, is it the output of the human gaze model? Or an example of a gaze from the human data? If it is plotting the output of a model of human attention, I would suggest changing the caption to ‘Human Model’, rather than ‘Human’, which implies human data is plotted. \n* **Section 4.2/Figure 3:**\n    * The figure plots a single example from two games, but the claims made appear to be broader. Beyond these examples (and examples in the appendix), could there be a metric for the number of objects the agents attend to? For example, the number of clusters/contiguous regions in the saliency maps produces by the agents? Computing a metric from more examples would help substantiate the point on the number of attended objects varying strongly as a function of $\\gamma$.\n    * The caption to Figure 3(b) mentions that “the human pays attention to an approaching enemy from the left and oxygen level at the bottom,” but the human panel only shows attention to the enemy on the left (and mildly to the agent itself). What supports the claim that the human pays attention to the oxygen bar? If additional gaze locations during this frame, it might be worth superimposing them somehow? As it is, the caption does not match the visual.\n    * Lines 215-217: the claim that attending to objects that matter in the long term is likely to be captured by covert attention, rather than overt attention — what is it based on? Is there any evidence for this claim, including negative evidence (e.g. not attending to the igloo in Frostbite for most of the gameplay, even though it is necessary to finish a level)?\n* Section 4.3/Figure 4: \n    * Minor narrative question: I found it mildly confusing that the discussion begins from the latter parts of the Figure — section 4.3 starts from discussing subfigures (d) and (e), and only then move to (a-c). It might be a little clearer to organize the subplots and the discussion in the same order.\n    * 4(d): is the implication of this figure that sometimes when the agents fail they attend to the same objects, and in other times when the agents fail, they attend to different objects? If I follow, it demonstrates that attention mismatch is neither necessary nor sufficient for failure.\n    * The caption mentions both significances at the 0.05 level (*) and at the 0.01 level (**), but all results reported in this figure are significant at the ** level. It could be clearer to omit the description of significance at the 0.05 level, then? Or am I missing the use of the single * somewhere? [this appears to only be used in Figure 5, in which case I might suggest mentioning it there]. \n    * Assuming I understand them, I find the empirical results reported in this section compelling — the degree to which agent attention and human attention match at failure states varies by games, in some games matching better in failure states, while in other games matching worse in failure states. However, I’m lacking any conclusion from this section, or any testable predictions or hypotheses. What is common between the games in which failure state attention is more similar, or less similar, or the games in which there appears to be no change in the distribution? Does this tell us something about what might be optimal in different games, or when matching human attention might be a better or worse predictor of success?\n* **Section 4.4/Figure 5:**\n    * Similar to the question for Figure 3 — a single example is plotted and followed with a more sweeping claim (similarities in Frostbite and Montezuma’s Revenge drop due to new objects the agents have never encountered). Is there evidence this statement generalizes beyond the plotted example? Was it made by inspecting all failure states? If it is important, I imagine it could be supported in a more quantitative way? e.g. locate the sprites of the new objects (matching single RGB pixels can be sufficient with Atari games, as they tend to use different colors), and compute the similarity between a saliency map computed from new objects only and the attention in these failure states.\n    * Lines 277-281: while I agree with the spirit of the conclusion, framing these results as locating ‘the obstacle’ between these agents and expert human-level performance seems rather ambitious. Perhaps ‘an obstacle? e.g. the other components [37] points to could prove equally important (if not more so). \n* **Section 4.5:**\n    * If Montezuma’s Revenge is excluded from further analysis because the other RL agents fail to learn it, why is it a good candidate for analysis using PPO?\n    * I find this section a little sparse — there is little doubt that the analyses performed for PPO _can_ be done for these other algorithms — but the lack of performing even a single one (for example, the analyses in figure 1, of the impact of learning on the similarity, or of the discount factor) does not provide any evidence that the result from analyzing other algorithms will be similar to the results with PPO. \n    * I understand the technical difficulty — performing these analyses would require either training the models, which is computationally expensive,  or receiving checkpoints at different intervals in training, which may not exist. However, as it stands, Section 4.5 offers little evidence that the results of the (interesting and meaningful) analyses on PPO will transfer to other algorithms, especially state-action value algorithms, due to the difference mentioned in the Appendix. \n* **Section 5:**\n    * Line 298: ”We have also identified further challenges in closing the performance gap between human experts and RL agents.” — I was a little surprised the text does not reiterate what these challenges are. \n    * Lines 311-314: “Our results indicate that RL agents first learn to attend to a few important objects like humans, then learn to attend to other objects.” This does not seem abundantly obvious from the results plotted in Figure 2, which as far as I can tell is the only attempt to visualize how many objects the agents attend to at different points in training? Is there any quantitative support for this point? See my question/suggestion regarding Figure 3 as well.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I agree with the authors’ assessment of the minimal direct social impact of their work and find their position on comparing humans and artificial agents compelling. \n\nHowever, the language of the abstract and the discussion frames their findings as regarding deep RL agents at large, while the evidence provided is limited to a single architecture. The results provided in Section 4.5 do not, to the best of my understanding, offer evidence that the conclusions from the analyses would generalize to other architectures; only that it would be feasible to perform the same analyses on other model architectures. I find this to be perhaps the work’s most significant limitation, and it is not discussed in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7-8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision has been reviewed and this paper has now been accepted.

----

After extensive discussions among SACs, ACs, and the program chairs, we have decided to conditionally accept this paper.  The crux of the issue is that the paper makes unsubstantiated causal claims about perception based on exploratory analyses (most notably in the final paragraphs of 4.3 and 4.4).  There was some debate around whether this constitutes a "fatal flaw," but it is certainly an issue that needs to be addressed.  The experiments and analyses should be reframed as exploratory first steps.  We hope that the authors will take into account feedback from the reviewers.

Specifically, in order to be accepted, the following conditions must be met:
- The experiments and analyses presented in the paper should be reframed as exploratory rather than causal or confirmatory.
- All unsubstantiated claims and interpretations of results (including those made in the final paragraphs of 4.3 and 4.4, but elsewhere in the paper too) should be removed.

----

The original meta-review for this copy of the paper follows:

This paper compares how humans and machines allocate attention while playing Atari games. The paper utilizes a dataset of human eye tracking measures, and compares a model of human gaze to the attention of RL agents. A number of factors were varied for the RL agents: amount of training, discount rates, whether game states are seen/unseen, etc. This type of work is valuable for interpreting deep RL models and inspiring new algorithms.

Reviewer opinions were initially split. There were a number of issues raised, but the biggest sticking point was the semi-frame-by-frame methodology used in the human Atari-HEAD dataset and whether that was comparable to the operation of the RL algorithms (see R-sG7Q for a summary of the objection). There was substantial back and forth on the issue, both in the public responses and private reviewer discussion. 

Although a full consensus was not reached, to summarize the spirit of the discussion, the method used by the authors is at least a plausible one and was justified. In my opinion, the deep RL systems are effectively processing the games frame-by-frame (or close to it), and thus aren't under much "time pressure" in the sense people are at 60 Hz. Thus, data from the human semi-frame-by-frame setting is well-positioned to inform the development of RL models (especially if the eye tracking data is higher quality in this setting). 

All things considered, I recommend the paper for acceptance.

In revisions, I hope that the authors will add some of their rebuttal regarding this issue to the paper, to further clarify this decision for other readers. I also agree with @hVFS that the choice of L2 distance in the saliency score (Eq 1) is a little strange, and isn't explained very clearly in the paper or rebuttal.

Overall, the analysis was well-thought out, thorough, and insightful. The open-source tools for comparing RL models with human attention will also be a valuable resource.
Decision: Accept (Poster)