Machine versus Human Attention in Deep Reinforcement Learning Tasks

1 Introduction . Researchers have devoted much effort to understanding deep neural networks ( DNNs ) . Since DNNs are partially inspired by the biological nervous systems , researchers have compared these neural networks with the human brain and sensory systems by asking two typical types of questions . The first one considers representation : How similar are the visual representations learned by DNNs and humans when performing the same tasks ? The second one concerns explainability : How do similarities and differences in the learned representations explain DNNs ’ performance on their tasks ? The first question motivates a seminal line of research that compares representations learned by DNNs with those learned by humans . In computer vision , a linear model that uses the feature map activations generated by a trained CNN can accurately predict neural activities in the early visual cortex , indicating that the two systems have learned similar visual representations [ 19 , 78 , 79 ] . In language learning , a similar connection is found between deep language models and cortical areas [ 31 , 34 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , this type of comparison has just emerged in decision-making research [ 44 , 13 ] , which motivates us to compare DNNs trained for deep reinforcement learning ( RL ) tasks with human decision-making . We ask : Do deep RL agents and humans agree on what visual features are important ? In other words , do agents pay attention to the same visual features as humans do ? The explainability question is motivated by the Explainable AI ( XAI ) for deep RL agents [ 29 , 2 , 55 ] . Deep RL has achieved many successes , but few of us fully understand these agents . These agents often learn a mapping from raw images to actions end-to-end where it is not clear why a particular decision is made . Our work addresses the explainability of these black-box models : Do RL agents make mistakes because they fail to attend to important visual features that matter for making the correct decision ? An expert human ’ s attention data could serve as a useful reference for identifying important visual features , which has been validated in object recognition tasks [ 50 ] . To answer these two questions , we situate our research in the domain of Atari games [ 6 ] . These games span a variety of dynamics , visual features , reward mechanisms , and difficulty levels for both humans and AIs . Two recent lines of research have made our work possible . On the one hand , a human attention dataset on Atari games has been collected using eye trackers [ 85 ] and convolutiondeconvolution networks have been shown to accurately predict human attention distributions [ 83 ] . On the other hand , tools have been introduced to generate visual interpretations of RL agents [ 23 ] so one can visualize an RL agent ’ s saliency map given an image , which is a topographically arranged map that assigns an importance weight to each image pixel and is often treated as the “ attention map '' of the RL agent . However , these tools can only provide qualitative results – we still need to manually inspect the saliency maps to interpret the performance . A useful comparison standard like human attention had not been considered hence systematic quantification was impossible . In this work , we present a novel study that compares the RL agent ’ s attention with both real and estimated human attention . We analyze how learning and hyperparameters of the RL algorithm affect the learned representations and saliency maps . We analyze failure and unseen states for RL agents and identify potential challenges to achieve human-level performance . We conclude by discussing insights gained for RL researchers in both cognitive science and AI . 2 Related Work . Human vs. machine attention in vision and language tasks . Human experts ’ gaze is very efficient and accurate for solving vision tasks . The peak angular speed of the human eye during a saccade reaches up to 900 degrees per second [ 59 ] . This allows humans to move their foveae to the right place at the right time to attend to important features [ 17 ] . Therefore , human expert ’ s gaze serves as a good standard in many vision-related tasks for evaluating machine attention , or as a learning target for training machine attention [ 57 , 84 ] . This approach is widely used in computer vision , see Nguyen et al . [ 51 ] for a review . For example , visual saliency researchers train DNNs to predict human visual attention . Saliency research is a well-developed field and we direct interested readers to recent review papers [ 7 , 10 , 26 , 9 ] . One such paper has compared visual saliency models with human visual attention [ 39 ] . In vision-related language tasks , such as image captioning and visual question answering , it was found that the saliency maps of DNN models are different from human attention [ 15 , 72 , 27 ] . Understanding and quantifying such differences have provided insights on the performance , especially in failure scenarios , of these vision-language models . Visual explanation for deep RL . Two classes of methods are widely used to generate visual interpretations of DNNs in the form of saliency maps : gradient-based and perturbation-based . Gradient-based methods compute saliency maps by estimating the input features ’ influence on the output using the gradient information [ 67 , 68 , 43 , 82 , 66 , 71 , 64 , 12 , 86 ] . These methods are for visualizing general DNNs but have been used to interpret deep RL agents [ 38 , 76 , 65 , 35 , 75 ] . Yet gradient-based saliency maps often lack physical meaning and could be difficult to interpret–they may highlight regions with no task-relevant objects or features [ 23 , 25 ] , thus are not used in our analyses . Perturbation-based methods alter parts of the input image and measure how much the output is affected by the change . Hence there have been different methods of altering the input [ 81 , 22 , 14 , 87 , 58 ] . These methods have been applied to Atari deep RL agents and can generate qualitatively meaningful saliency maps [ 23 , 33 , 24 , 56 ] . However , without human attention as a reference , it is difficult to quantitatively analyze these saliency maps , which motivates our work . There are also methods that change the architecture of the deep RL network by augmenting it with an explicit artificial attention module , so that one can directly access its attention map [ 49 , 80 , 48 ] . Researchers have taken this approach and compared RL agents ’ attention with human attention [ 52 ] . However , these methods do not apply to general deep RL algorithms since they need to modify the original network architectures and retrain the new ones . 3 Method . We now discuss methods for modeling human attention , training RL agents , and extracting attention information from the trained RL agents . We then discuss how we select data for comparing human versus RL attention , and define the comparison metrics . Human attention data and model We use human expert gaze data from Atari-HEAD dataset [ 85 ] . The original game runs continually at 60Hz [ 6 ] , a speed that is challenging even for professional gamers . Human eye movements were rushed and inaccurate at this speed and hence could not serve as a useful reference . In [ 85 ] , however , the human data were collected in a semi-frame-by-frame mode . Without changing the action manipulation or the reward structures of the games , this design mimicked the frame-by-frame processing of RL agents for the human players . The players were allowed enough time to process the visual input , producing higher-quality data with more precise attention allocation and action execution . These advantages led to world expert-level performance in [ 85 ] –about 35x better than the human performance reported in previous deep RL literature [ 46 , 28 ] . To get human saliency maps for the image states generated by RL agents , we need an accurate human attention model . RL agents make many more mistakes than expert humans and can not reach the late stages of the games . This creates a state distribution that does not match human data , which contains mostly good states . Having a human attention model allows us to perform comparisons in states encountered by RL agents only , which is especially important when we later analyze RL agents ’ failure states . Predicting human attention is a challenging task in Atari games . The human gaze is rarely on the player ’ s avatar or contingency regions [ 6 ] so simple object detectors would not work . Human players often select and focus on a few among multiple visually identical objects , and divide their attention if multiple objects are relevant for decision making . Recently , variants of convolution-deconvolution network models have achieved the best results on predicting human attention [ 42 , 83 , 53 , 16 ] . Here we followed their approach and trained gaze models from ground truth human gaze data . Given an image , the models produce a heatmap of the same dimension that predicts human attention distribution . These models have been evaluated and described in detail in previous studies ( e.g. , [ 83 ] ) , so we leave out the details in Appendix 1 . As a control analysis , we include two attention models in addition to the human gaze . The first one captures the motion information , measured by Farneback optical flow between two consecutive images [ 20 ] . The second model captures salient low-level image features , including color , orientation , and intensity ( weighted equally ) , computed by the classic Itti-Koch saliency model [ 32 ] . Implementation details of these models can be found in Appendix 1 . Reinforcement learning agent and attention model As a case study , we use a popular deep RL algorithm named Proximal Policy Optimization ( PPO ) [ 63 ] with default hyperparameters [ 30 ] . For each experimental condition ( discussed later ) , we train 5 models with different random seeds to capture the variance in training and ensure reproducibility . For the Atari gaming environment , we use the basic version that has no frame skipping and no stochasticity in action execution ( NoFrameskip-v4 version ) . In order to capture variance in training and ensure reproducibility , we select six popular Atari games instead of using all Atari games and train 540 agents in total , with the same hyperpareameters except for random seeds and discount factors ( see Section 4 ; 300 GPU days on GeForce GTX 1080/1080 Ti ) . We use a perturbation-based method [ 23 ] to extract attention maps from PPO agents , which has been validated by several subsequent studies [ 25 , 65 ] . The algorithm takes an input image I and applies a Gaussian filter to a pixel location ( i , j ) to blur the image . This manipulation adds spatial uncertainty to the region around and produces a perturbed image Φ ( I , i , j ) . Denote the learned policy as π and the inputs to the final softmax activation as πu ( I ) for image I ( i.e . the last latent representation ) . A saliency score for this pixel ( i , j ) can be defined as how much the blurred image changes the latent representation πu ( I ) in Euclidean space : Sπ ( i , j ) = 1 2 ‖πu ( I ) − πu ( Φ ( I , i , j ) ) ‖2 . ( 1 ) Intuitively , Sπ ( i , j ) describes how much removing information from the region around location ( i , j ) changes the policy [ 23 ] . In other words , a large Sπ ( i , j ) indicates that the information around pixel ( i , j ) is important for the learning agent ’ s decision making . Instead of calculating the score for every pixel , [ 23 ] found that computing a saliency score for pixel i mod 5 and j mod 5 produced good saliency maps at lower computational cost for Atari games . The final saliency map P is normalized as P ( i , j ) = Sπ ( i , j ) ∑ i , j Sπ ( i , j ) . Comparison metrics Next , we compile a set of game images to compute human and RL saliency maps . For each game , we let a trained PPO agent ( with default hyperparameters ) play the game until terminated , and uniformly sampled 100 images from the recorded trajectory . We will refer to this set of images as the standard image set . We then define two metrics for comparing saliency maps when using RL data : Pearson ’ s Correlation Coefficient ( CC ) and Kullback-Leibler Divergence ( KL ) . Let Q denote the human saliency map predicted by the human attention network . CC is between −1 and 1 captures the linear relation between two distributions Q and P : CC ( P , Q ) = σ ( P , Q ) σP × σQ ( 2 ) where σ ( P , Q ) denotes the covariance , and σP and σQ are the standard deviations of P and Q respectively . CC penalizes false positives and false negatives equally . However , we may not want to penalize the RL agent if it attends to regions that the human gaze model is not currently focused on . The human gaze data only reveals the “ overt ” attention , and humans can still pay “ covert ” attention to entities in the working memory [ 54 , 62 ] . In other words , being attended to by the human gaze is a sufficient ( but not necessary ) condition for the features to be important . Thus we need a second metric that penalizes the agent only if it fails to pay attention to human attended regions , or equivalently , a metric that is sensitive to false negatives if we treat human attention as the ground truth . KL is an ideal candidate in this case [ 9 ] : KL ( P , Q ) = ∑ i ∑ j Q ( i , j ) log ( + Q ( i , j ) + P ( i , j ) ) ( 3 ) where is a small regularization constant ( chosen to be 2.2204e-16 [ 9 ] ) and determines how much zero-valued predictions are penalized .

Summary: This work proposes to use human gaze data and an attention model derived from it to compare and contrast human attention and attention saliency maps extracted from PPO deep RL agents on Atari Learning Environment games. The authors analyze the effect of learning, discovering that learning improves similarity between agent attention and human attention. The authors evaluate how the discount factor contributes to similarity, finding that attentional similarity and performance peak simultaneously, with fairly high discount factors. The authors also perform some focused analyses of how attention varies in failure states for these artificial agents, and how attention behaves when attempting to generalize to unseen data from the same games.