Machine versus Human Attention in Deep Reinforcement Learning Tasks

Abstract: Deep reinforcement learning (RL) algorithms are powerful tools for solving visuomotor decision tasks. However, the trained models are often difficult to interpret, because they are represented as end-to-end deep neural networks.  In this paper, we shed light on the inner workings of such trained models by analyzing the pixels that they attend to during task execution, and comparing them with the pixels attended to by humans executing the same tasks. To this end, we investigate the following two questions that, to the best of our knowledge, have not been previously studied. 1) How similar are the visual representations learned by RL agents and humans when performing the same task? and, 2) How do similarities and differences in these learned representations explain RL agents' performance on these tasks? Specifically, we compare the saliency maps of RL agents against visual attention models of human experts when learning to play Atari games. Further, we analyze how hyperparameters of the deep RL algorithm affect the learned representations and saliency maps of the trained agents. The insights provided have the potential to inform novel algorithms for closing the performance gap between human experts and RL agents.


Reviews and author responses:

 We kindly point the reviewer to Sections 4 and 5 where we made the conclusions and stated the contributions of this work. We concluded that there is a significant correlation between the agents’ performance and their attention similarity with humans, both across games and across architectures. The tasks we selected span a wide range of perceptual structures and learning difficulties for the RL agents, despite the limitation in resources. We hope our proposed method helps identify the appropriate hyperparameters, diagnoses RL failures in critical states, and also promotes the usage of human attention data in RL training.

 Perception is indeed a core component of performance, but so is policy learning. Our analyses in Section 4.3 help us distinguish tasks that are difficult for the agents in the above two aspects: (1) if the agents have _less_ similar attention to humans when they fail, they are more likely to have bad perception and thus the task is more perceptually difficult; (2) if the agents have _more_ similar attention to humans when they fail, they are more likely to have learned a suboptimal policy _despite_ good perception. This failure case analysis does not conflict with the results in Section 4.1 and Section 4.5, where, in general, better perception correlates with better performance, since good perception contributes to successful policy learning. 

As for your suggestion on controlled experiments--we did train five agents on the same task with different random seeds under the same training protocol, and have confirmed that each set of agents yield very similar performance and attention maps. We would consider the experiments in Section 4.1 to be the experiment you suggested: The not fully-trained agents have poorly correlated perception, and the fully trained agents have highly correlated perception. In this case, we do see that performance has a strong correlation with attention similarity. We hope this answer resolves your confusion.


In this work RL agent's attention saliency maps are compared to those of a human player (via a model trained to predict human attention from gaze). Comparison of the maps during learning and in the failure modes allow to make some interesting observations that can lead to curious interpretation about the role of attention in RL agents.     106: Why one needs to preduct human attention from gaze? Wouldn't gaze provide that information directly? And if not, then what is being used as the ground truth to train the gaze->attention model? Why not use that ground truth as the one that is being compared to RL agent's attention saliency map? In order words, it seems that the current pipeline is
gaze map --model--> human attention map <--comparison--> RL attention map
why not direclty
gaze map <--comparison--> RL attention map
Is it because that way you would be limited only to the states that exist in the [ref:81] dataset? Would that be insufficient?

Figure 1: Maybe it's just me but the labels are a bit confusing: "Human CC" is CC between human and RL, right? Why it is called "Human" then? Perhaps "CC(Human, RL)", "CC(Motion, RL)", etc would be more clear.

Overall the question that is being explored in this paper is one high significance to the general debate of the differences and similarities between biological and artificla systems of vision. This work does not complete the full analysis that could allows us to make strong claims about such differences or similarities, but it provides an example of methodological approach and some curious observations.

The section 4.5 was supposed to address the issue of comprehensiveness, but perhaps fails to achieve that goal. The fact that there is a relatively strong correlations, this correlation is insufficient to assume that the results of a detailed analysis on other algorithms will be the same as with PPO. This does not invalidate the aproach itself, but leaves the reader wondering about the conclusion, the lessons learned and the exact contribution of this work.

That being said I would not mind seeing this work presented at NeurIPS as it provides a first step in a quite interesting directions and adds to the volume of knowledge on comparison between biological and artificial learning systems.

A note regarding "We agree that real-world tasks would be more impactful. However, due to the difficulty of applying RL algorithms in the real world, a comprehensive analysis (e.g., failure analysis) is infeasible" that was mentioned in the "Submission History - Improvements Made". There are online systems such as https://gym.offworld.ai that allow to run algorithms on real-world RL systems remotely. If a real-world evaluation would help advance your results you might consider using those.

--------- UPDATE ----------
Due to the considerations reflected in my response (see below) I am lowering the score to "5: Marginally below the acceptance threshold". The authors do address some limitations of their work and do not overestimate their claim.

 Thank you for you response and clarifications of the points raised.

I have read the paper once again to gain a fresh and overall perspective. At this time I am inclined to re-evaluate this contribution as being below NeurIPS acceptance threshold.

I still am fond of the idea to compare human and machine attention on the same task and value the potential explanatory power of this method for better understanding of RL models. However the paper lacks the clear outcome by the end it remains unclear whether those attentions are indeed similar or the correlations are too low to claim that. Perhaps a significantly more extensive tensing under various conditions would highlight the exact value and knowledge this methods can give us, but in its present form the paper provides the idea, but does not follow that up with a rigorous evaluation and clear conclusions.

 I thank the authors for taking the time to respond to my comments about Section 4.3. However, I am still not seeing the takeaway or the actionable message of this section or result. 

The author's response says that the Section 4.3 data: ".. don't show consistent trends among all games -- but this is exactly what we want to empathize (sic). From the inconsistencies we can distinguish tasks that are perceptually difficult for RL agents vs. tasks that are not. This helps us understand why the same RL algorithm succeeded on some games and failed on the others."
And then shortly thereafter says: "...perfect perception does not imply good performance -- we simply want to bring this discovery into the discussion -- that perceptual errors are an issue for some of these games and need to be addressed in developing better RL algorithms."

This again seems to be building a case on speculation that hasn't been directly tested, which is at the heart of the original comment. The authors suggest that perception is a core component of performance, which is perfectly reasonable, but it is not clear that scoring perception by human-correlated saliency is meaningful or useful. It is not clear that, with human-level perception, we would see high performance. If the authors directly compared two agents-- poorly correlated perception vs. highly correlated perception -- with the same core RL algorithm on the same set of tasks, we might be able to say something about how the aligned saliency maps relate to performance. As it stands, the experiment leaves too many variables uncontrolled to draw a meaningful conclusion.

 Thank you for your feedback! Regarding your questions and comments:

> I was wondering why the human attention data set was collected in the first place, of not to compare it to study DNNs?

It was originally collected as a general-purpose dataset for all human-centered AI methods such as imitation learning. 

> In the sentence, "RL agents mistakes are related to failing to pay attention to the CORRECT visual targets", I would like the authors to define what they mean by "correct" in this context. How would they define the ground truth of the "correct" visual target to pay attention to? Would a concept like "behaviorally relevant" be closer to what they meant?

Here "correct visual targets" refers to the targets being attended by expert humans, e.g., the ball in Fig.4d Breakout. These targets are often a subset of all behaviorally relevant targets on the screen.

> Overall, the text contains a few instances of phrases that are somewhat unclear. For example, what do you mean "they lack physical meaning"? What are "qualitatively meaningful saliency maps"?

These are high-level conclusions drawn from previous RL interpretability research. We will make reference to these works (e.g., [21,23]) that are more prominent. For example, a gradient-based method may generate a saliency map that highlights a region that has no task-relevant objects or features, which is not considered as qualitatively meaningful by humans.

> It is also unclear what they authors mean by "world expert level performance". Why did the human participants in [81] reach this level of performance (assuming they were not all world experts)? What is the world expert level based on, is it a different task, maybe the speeded-up version? This paragraph needs to be clarified.

You are right that these humans are not world experts. We explained in lines 92-95 the reason that they achieved such performance is due to slowed game speed. This design allowed humans to make accurate eye movements and correct decisions -- which are very important for our study since we are using human data as a standard for comparison.

> It is unclear what the author mean by "predict human attention from gaze data"? If attention is the target, and gaze data is the predictor, what is the ground truth? Do the authors mean they trained the network to predict gaze data on the held-out set from gaze data on the training set, which both are meant as approximations of attention?

Here we are simply performing the traditional saliency prediction task. The network is trained to predict human gaze/attention given an image in a supervised learning setting. The ground truth is the actual human gaze data. We will rephrase this sentence as "predict human attention given an image, trained using human gaze data as ground truth" to avoid confusion.

> "after training, human and RL saliency maps are more positively correlated" Is the claim of an increase based on statistical evidence? For example, did the authors calculate a correlation with time of the decreasing difference, or a slope over time in a regression model?

The Pearson correlation coefficients of attention similarity with time are: CC - r(8) = 0.992, p < 0.01; KL - r(8) = 0.987, p < 0.01.

> My biggest concern, however, is not a technical one, but conceptual in nature: I am wondering if the study really compares attention. The way attention is measured in the networks is by checking how much it would change the policy when certain pixels are blurred. So what the study might actually shows is that human pay attention to what should affect their policy, rather than showing that a DNN is paying attention to anything - after all the authors are just measuring the effects pixels have on the policy, i.e., pixel-induced behavioral differences.

We agree that whether pixel-induced behavioral differences should be regarded as the best "attention" mechanism of DNNs is subject to debate. However, for decision-making DNNs, its "attention" mechanism should highlight pixels that are important for the behavior, i.e., removing / blurring the pixels would drastically change the behavior. Hence the method is widely regarded as the attention mechanism in XRL research (see survey papers such as Reinforcement Learning Interpretation Methods: A Survey).

> How is the choice of hyperparameters related to agents' game performance? Can the difference in "attention" simply be explained by the quality of the behavioral policy? That is, wouldn't a better policy automatically be affected more by the pixels that contain the relevant information, just by virtue of using the relevant information to guide its behavior? This view is also supported by the fact that better agents had higher similarity scores to humans, who also showed better behavior.

This is the question we tried to answer in section 4.2. We know from previous works that tuning hyperparameters like the discount factor will affect performance (Fig.1(b) blue line). But the question remains why this is the case. Section 4.2 shows that appropriate discount factor choice will result in a proper planning horizon (line 224-230), which might be the reason.

> Are there any statistics on the error analysis? It would be necessary to back on qualitative claims with statistical analyses.

We show aggregated statistics for all games (with variance and significance level, Fig 4(a)),  and distributions of two example games (b,c). Please let us know if you would like us to report any other statistical analyses.

> I found the bottom-up vs top-down discussion quite interesting, particularly its evolution over time.
The paragraph about alternative agent architectures is not satisfying. The authors did NOT show that anything generalizes. They only provide a correlation with a new measures, and do not report any of the previous analyses. As it is, it does not support the conclusion that the pattern arise for other agent architectures. If the relevant tests have been conducted, they need to be reported here.

We are simply suggesting that the positive correlation between model performance and similarity with human attention generalizes to other deep RL algorithms. We are not claiming that all previous results on learning, discount factors, etc, generalize to these RL algorithms. As we have discussed in line 119, a complete set of experiments on one algorithm requires training 630 agents, which is not feasible for all these algorithms.

> Limitations And Societal Impact:
> The authors do not talk sufficiently about the limitations of their work.

We hope our answers above help address some of the limitations you mentioned. For the rest, we will make sure we discuss them sufficiently in the revision.

The paper has lots of details, so it's understandable that the reviewer came away with a few misconceptions. However, we maintain that most, if not all, of the reviewer's concerns are already addressed in our text. 


 Thank you for your constructive feedback! Regarding your questions and comments:

> Section 4.3 concludes by saying: "Similarity measurements with human attention can help identify and interpret the failures made by RL agents." However, Figure 4 and the data from the section seem to indicate that no conclusions can be drawn, as the results are inconsistent. 

The results on failure and unseen data don't show consistent trends among all games -- but this is exactly what we want to empathize. From the inconsistencies we can distinguish tasks that are perceptually difficult for RL agents vs. tasks that are not. This helps us understand why the same RL algorithm succeeded on some games and failed on the others. 

> Section 4.3 concludes that mistakes by RL agents are more likely related to perceptual errors in certain games. Again, this is not supported by the data, where perceptual errors do not explain the failure states of certain agents. There is no guarantee that, with perfect perception, the agents would have performed well in these games. Perceptual errors may be related to the decision-making errors, but the data here is not conclusive and does not appropriately target this research question.

This section only presents a correlation between mistakes and perceptual errors. And you are right that perfect perception does not imply good performance -- we simply want to bring this discovery into the discussion -- that perceptual errors are an issue for some of these games and need to be addressed in developing better RL algorithms.

> The paper does not adequately address the failings of attention as a mechanism for interpretability, though this is considered as a motivator for the work. Attention has been demonstrated to be a poor proxy for what is truly "important" or relevant, as even random networks. For reference, check "Sanity Checks for Saliency Maps" Adebayo et al. 2018: https://papers.nips.cc/paper/2018/file/294a8ed24b1ad22ec2e7efea049b8737-Paper.pdf

We understand your concern and are familiar with the reference. Adebayo et al.'s results are limited to computer vision tasks and gradient-based saliency methods. That is why methods such as the one we use [21] were developed for XRL research and they performed reasonably well (e.g., see related work of [21] or Reinforcement Learning Interpretation Methods: A Survey by Alharin et al.).

> The checklist says that code and data are available, but the supplementary material is only a PDF.
> The appendix says that a gaze-prediction video is included in a multimedia appendix, but this does not seem to exist.

Our apology -- we removed these since they are already public and do not conform with anonymity requirements.


 Thank you for your constructive feedback! Regarding your questions and comments:
> 106: Why one needs to predict human attention from gaze?...

You raised a very important point here. We tried to explain this issue called “state distribution mismatch” in line 96-101. The human data and RL data are different due to the difference in their policies. Having a gaze prediction model (trained on human gaze data) allows us to compare attention in new states generated by the RL agent, so we are no longer limited to human states in [81]. We also conducted experiments using the ground truth human gaze (without prediction) and the results show a similar trend as presented in the main results (line 334 and whole Appendix 7).

> Figure 1: Maybe it's just me but the labels are a bit confusing: "Human CC" is CC between human and RL, right? Why it is called "Human" then? Perhaps "CC(Human, RL)", "CC(Motion, RL)", etc would be more clear.

That is a good suggestion and we will do as suggested.

> The section 4.5 was supposed to address the issue of comprehensiveness...

Thank you for pointing this out. We will weaken our claim and emphasize that this correlation does not necessarily imply that the results of a detailed analysis on other algorithms will be the same as with PPO.

> There are online systems such as https://gym.offworld.ai that allow to run algorithms on real-world RL systems remotely.

Thank you for suggesting this useful resource. Using simulators for more realistic RL tasks and human studies is indeed a promising direction for future work.


 We will add PPO results to Appendix 6 Figures. Thank you for your encouraging comments!

- This paper presents an empirical study of machine vs. human attention in Deep RL on Atari.  Specifically they examine how the attention of an CNN agent correlates with human attention
- They find that human attention and RL attention correlate well and this correlation improves as agent performance improves.  For attention methods like low-level image feature salience, however correlation with RL attention reduces as performance increases, suggesting that RL attends to higher-level features in the image as training progresses.
- The examine how the discount factor changes attention, specifically a lower discount factor helps RL learn to attend to similar things as the human experts.
-  Finally they show that in failure states, RL attention often deviates from human attention -- although not always.
  - Strengths
    - Experiments are interesting and well done
    - The paper is well-written and the discussion is section is useful as it provides some (possibly) useful future directions for researchers to improve RL algorithms.
    - The finding that attention is less correlated with human attention in failure states is quite interesting
    - The supplement provides analysis of addition RL algorithms, showing that the trend that RL performing increasing results in better correlation with human attention generalizes to other RL algorithms.
        - Note: Having PPO on those plots would be quite useful.
- Weaknesses
    - Ideally a set of tasks other than Atari would have been used as Atari is largely solved.  However, neither human attention datasets nor human experts exist for similar things like ProcGen and DMControl and I believe this paper is useful regardless.
- Overall
    - I found this paper interesting and well-written and I believe it would be of interest to the NeurIPS audience.
 Yes, limitations are well discussed.

The study aims to compare human and artificial attention in the realm of decision making. This research supports understanding of the function (and potential malfunction) of neural networks, and also aims to shed light on potential mechanisms of improved training of artificial agents. Human attention is operationalized as gaze data (which pixels do the eyes fixate?), and artificial agent attention is operationalized as the blurring of which pixels has the largest effects on behavior policy.   - First, I think this kind of work is of great value for artificial agent analysis research because better understanding of the agents we are working with is often crucial, and often missing. 
- I does not seem as relevant to claim repeatedly be "the first" in this particular area of research - if the study is conducted well and of high relevance, claims of priority come second.
- I was wondering why the human attention data set was collected in the first place, of not to compare it to study DNNs?
- In the sentence, "RL agents mistakes are related to failing to pay attention to the CORRECT visual targets", I would like the authors to define what they mean by "correct" in this context. How would they define the ground truth of the "correct" visual target to pay attention to? Would a concept like "behaviorally relevant" be closer to what they meant?
- Overall, the text contains a few instances of phrases that are somewhat unclear. For example, what do you mean "they lack physical meaning"? What are "qualitatively meaningful saliency maps"?
- It is also unclear what they authors mean by "world expert level performance". Why did the human participants in [81] reach this level of performance (assuming they were not all world experts)? What is the world expert level based on, is it a different task, maybe the speeded-up version? This paragraph needs to be clarified. 
- It is unclear what the author mean by "predict human attention from gaze data"? If attention is the target, and gaze data is the predictor, what is the ground truth? Do the authors mean they trained the network to predict gaze data on the held-out set from gaze data on the training set, which both are meant as approximations of attention?
- "after training, human and RL saliency maps are more positively correlated" Is the claim of an increase based on statistical evidence? For example, did the authors calculate a correlation with time of the decreasing difference, or a slope over time in a regression model?
- My biggest concern, however, is not a technical one, but conceptual in nature: I am wondering if the study really compares attention. The way attention is measured in the networks is by checking how much it would change the policy when certain pixels are blurred. So what the study might actually shows is that human pay attention to what should affect their policy, rather than showing that a DNN is paying attention to anything - after all the authors are just measuring the effects pixels have on the policy, i.e., pixel-induced behavioral differences.
- How is the choice of hyperparameters related to agents' game performance? Can the difference in "attention" simply be explained by the quality of the behavioral policy? That is, wouldn't a better policy automatically be affected more by the pixels that contain the relevant information, just by virtue of using the relevant information to guide its behavior? This view is also supported by the fact that better agents had higher similarity scores to humans, who also showed better behavior. 
- Are there any statistics on the error analysis? It would be necessary to back on qualitative claims with statistical analyses. 
- I found the bottom-up vs top-down discussion quite interesting, particularly its evolution over time.
- The paragraph about alternative agent architectures is not satisfying. The authors did NOT show that anything generalizes. They only provide a correlation with a new measures, and do not report any of the previous analyses. As it is, it does not support the conclusion that the pattern arise for other agent architectures. If the relevant tests have been conducted, they need to be reported here.

Overall, the work seems to too preliminary for publication at this stage, given the lack of statistical measures in some places, and vague wording and explanations. More crucially though, the authors need to define what they mean by "attention", and how this concept might differ between humans (gaze) and DNNs (perturbation). If their operationalization of "attention" indeed reflects information gathering in one case (human), but the accuracy of a behavioral policy in the other (DNN), comparing them might not be the right way to understand DNNs better.  The authors do not talk sufficiently about the limitations of their work. 

The paper introduces an analysis of the relationship between RL agent visual attention and human visual attention for Atari games. After training pixel-based RL agents, the work introduces comparisons at several stages of training, showing increasing correlation with a model of human visual attention. The paper then analyzes instances of matched or mismatched correlation, and provides a failure analysis of the RL agent’s behavior based on the visual attention synchronization. Extensive experimental details and further analysis is included in the appendix.  

The paper is clearly written and communicates its contributions, motivations, methods, experiments, and data all clearly. The research question is interesting, though ultimately the investigations presented are shallow and the conclusions are inconsistent. For example, Section 4.3 concludes by saying: "Similarity measurements with human attention can help identify and interpret the failures made by RL agents." However, Figure 4 and the data from the section seem to indicate that no conclusions can be drawn, as the results are inconsistent. Similarly, the paper concludes that pre-training to mimic human attention may be a useful prior for RL agents. However, this is not supported clearly by the data in the paper, and no experiments are run to explicitly examine this hypothesis. 

Clearly stating research questions and metrics for each section/question up-front, and qualifying the contributions/claims made with the data would significantly improve this paper.

Strengths:
 * The paper is well-motivated and provides a strong incentive to perform an analysis of human-to-RL-agent visual attention.
* Each section is well-written and self-contained. Data and analyses are clearly written, and each section provides concrete takeaways for future research.
* There is an overwhelming amount of experimentation in this paper (and in the appendix). There was clearly a great deal of engineering effort and analysis involved to establish takeaways and patterns in the data.

Weaknesses:
* The paper’s takeaways are not often supported by the data. In fact, the data are often contradictory and not in agreement. Rather than providing an in-depth analysis of why these contradictions appear (such as by manipulating RL-agent attention and examining the effects on task performance, or by pre-training an RL-agent with the human-attention model), the paper often picks one of the possible conclusions, presents it as the ground-truth takeaway, and moves on.
* Section 4.3 concludes that mistakes by RL agents are more likely related to perceptual errors in certain games. Again, this is not supported by the data, where perceptual errors do not explain the failure states of certain agents. There is no guarantee that, with perfect perception, the agents would have performed well in these games. Perceptual errors may be related to the decision-making errors, but the data here is not conclusive and does not appropriately target this research question.
* Section 4.4 concludes that human-level performance must be preceded by human-level attention. While this is loosely supported by the findings in Section 4.1, again the research question is not being targeted by the data. Pre-training a visual model on human attention might be one way to answer this question, but the current accuracy for seen/unseen states does not seem to support this conclusion.

Other notes:
* The checklist says that code and data are available, but the supplementary material is only a PDF.
* The appendix says that a gaze-prediction video is included in a multimedia appendix, but this does not seem to exist.

 The paper does not adequately address the failings of attention as a mechanism for interpretability, though this is considered as a motivator for the work. Attention has been demonstrated to be a poor proxy for what is truly "important" or relevant, as even random networks. For reference, check "Sanity Checks for Saliency Maps" Adebayo et al. 2018: https://papers.nips.cc/paper/2018/file/294a8ed24b1ad22ec2e7efea049b8737-Paper.pdf

Meta-review: UPDATE: The revision has been reviewed and this paper has now been accepted.

----

After extensive discussions among SACs, ACs, and the program chairs, we have decided to conditionally accept this paper.  The crux of the issue is that the paper makes unsubstantiated causal claims about perception based on exploratory analyses (most notably in the final paragraphs of 4.3 and 4.4).  There was some debate around whether this constitutes a "fatal flaw," but it is certainly an issue that needs to be addressed.  The experiments and analyses should be reframed as exploratory first steps.  We hope that the authors will take into account feedback from the reviewers.

Specifically, in order to be accepted, the following conditions must be met:
- The experiments and analyses presented in the paper should be reframed as exploratory rather than causal or confirmatory.
- All unsubstantiated claims and interpretations of results (including those made in the final paragraphs of 4.3 and 4.4, but elsewhere in the paper too) should be removed.

----

The original meta-review for this copy of the paper follows:

Given that three reviewers have voted to reject and the fourth reviewer has (privately) expressed some concerns about the paper in the reviewer discussion, I will be recommending reject. 

Central to the issue with this paper is reviewer YjzP's comment: "The authors suggest that perception is a core component of performance, which is perfectly reasonable, but it is not clear that scoring perception by human-correlated saliency is meaningful or useful". the author's response that "(1) if the agents have less similar attention to humans when they fail, they are more likely to have bad perception and thus the task is more perceptually difficult" seems to be a correlative vs. causal claim that remains to be shown.

The paper does not address limitations of saliency maps, or control for those limitations as null hypotheses in their experiments - the papers assume that the saliency model for human gaze and the saliency model for RL agent do explain agent behavior. The authors claim that they use a more explainable saliency method (perturbation-based), but the issue goes beyond saliency vs. perturbation. The paper "Axiomatic Attribution for Deep Neural Networks" (https://arxiv.org/abs/1703.01365) points some of these issues out. The paper is also missing an important citation to a recently published paper, "Exploratory not Explanatory: Counterfactual Analysis of Saliency Maps for Deep RL" (ICLR 2020, https://arxiv.org/pdf/1912.05743.pdf), with a more rigorous analysis on the pitfalls of saliency (Section 3.2). Along these lines, all four reviewers and myself included would like to see more "counterfactual evaluation of claims", and "falsifiable hypotheses from saliency maps" for the claims made in the paper.

An example of "falsifiable hypothesis": Although CC/KL is highly correlated with game score (>0.8), this does not imply that the agent is attending in the same way as the agent. A test example of how this might occur: suppose the RL agent learns to attend to 20 different things, 1 of which is what the human attends to. At the start of training, RL agent does not attend to the 1 thing, and performance is low. After training, agent attends to the 1 thing (inducing a mildly weak correlation with human attention), and scores are high. The correlation is still weak, but is slightly higher. Then you measure a high correlation between the CC/KL metric and performance, which has essentially magnified the effect of the weak correlation into a strong one on performance. This is one null hypothesis for the results that the authors did not rule out. 



Decision: accept