Machine versus Human Attention in Deep Reinforcement Learning Tasks

1 Introduction . Researchers have devoted much effort to understanding deep neural networks ( DNNs ) . Since DNNs are partially inspired by the biological nervous systems , researchers have compared these neural networks with the human brain and sensory systems by asking two typical types of questions . The first one considers representation : How similar are the visual representations learned by DNNs and humans when performing the same tasks ? The second one concerns explainability : How do similarities and differences in the learned representations explain DNNs ’ performance on their tasks ? The first question motivates a seminal line of research that compares representations learned by DNNs with those learned by humans . In computer vision , a linear model that uses the feature map activations generated by a trained CNN can accurately predict neural activities in the early visual cortex , indicating that the two systems have learned similar visual representations [ 19 , 78 , 79 ] . In language learning , a similar connection is found between deep language models and cortical areas [ 31 , 34 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , this type of comparison has just emerged in decision-making research [ 44 , 13 ] , which motivates us to compare DNNs trained for deep reinforcement learning ( RL ) tasks with human decision-making . We ask : Do deep RL agents and humans agree on what visual features are important ? In other words , do agents pay attention to the same visual features as humans do ? The explainability question is motivated by the Explainable AI ( XAI ) for deep RL agents [ 29 , 2 , 55 ] . Deep RL has achieved many successes , but few of us fully understand these agents . These agents often learn a mapping from raw images to actions end-to-end where it is not clear why a particular decision is made . Our work addresses the explainability of these black-box models : Do RL agents make mistakes because they fail to attend to important visual features that matter for making the correct decision ? An expert human ’ s attention data could serve as a useful reference for identifying important visual features , which has been validated in object recognition tasks [ 50 ] . To answer these two questions , we situate our research in the domain of Atari games [ 6 ] . These games span a variety of dynamics , visual features , reward mechanisms , and difficulty levels for both humans and AIs . Two recent lines of research have made our work possible . On the one hand , a human attention dataset on Atari games has been collected using eye trackers [ 85 ] and convolutiondeconvolution networks have been shown to accurately predict human attention distributions [ 83 ] . On the other hand , tools have been introduced to generate visual interpretations of RL agents [ 23 ] so one can visualize an RL agent ’ s saliency map given an image , which is a topographically arranged map that assigns an importance weight to each image pixel and is often treated as the “ attention map '' of the RL agent . However , these tools can only provide qualitative results – we still need to manually inspect the saliency maps to interpret the performance . A useful comparison standard like human attention had not been considered hence systematic quantification was impossible . In this work , we present a novel study that compares the RL agent ’ s attention with both real and estimated human attention . We analyze how learning and hyperparameters of the RL algorithm affect the learned representations and saliency maps . We analyze failure and unseen states for RL agents and identify potential challenges to achieve human-level performance . We conclude by discussing insights gained for RL researchers in both cognitive science and AI . 2 Related Work . Human vs. machine attention in vision and language tasks . Human experts ’ gaze is very efficient and accurate for solving vision tasks . The peak angular speed of the human eye during a saccade reaches up to 900 degrees per second [ 59 ] . This allows humans to move their foveae to the right place at the right time to attend to important features [ 17 ] . Therefore , human expert ’ s gaze serves as a good standard in many vision-related tasks for evaluating machine attention , or as a learning target for training machine attention [ 57 , 84 ] . This approach is widely used in computer vision , see Nguyen et al . [ 51 ] for a review . For example , visual saliency researchers train DNNs to predict human visual attention . Saliency research is a well-developed field and we direct interested readers to recent review papers [ 7 , 10 , 26 , 9 ] . One such paper has compared visual saliency models with human visual attention [ 39 ] . In vision-related language tasks , such as image captioning and visual question answering , it was found that the saliency maps of DNN models are different from human attention [ 15 , 72 , 27 ] . Understanding and quantifying such differences have provided insights on the performance , especially in failure scenarios , of these vision-language models . Visual explanation for deep RL . Two classes of methods are widely used to generate visual interpretations of DNNs in the form of saliency maps : gradient-based and perturbation-based . Gradient-based methods compute saliency maps by estimating the input features ’ influence on the output using the gradient information [ 67 , 68 , 43 , 82 , 66 , 71 , 64 , 12 , 86 ] . These methods are for visualizing general DNNs but have been used to interpret deep RL agents [ 38 , 76 , 65 , 35 , 75 ] . Yet gradient-based saliency maps often lack physical meaning and could be difficult to interpret–they may highlight regions with no task-relevant objects or features [ 23 , 25 ] , thus are not used in our analyses . Perturbation-based methods alter parts of the input image and measure how much the output is affected by the change . Hence there have been different methods of altering the input [ 81 , 22 , 14 , 87 , 58 ] . These methods have been applied to Atari deep RL agents and can generate qualitatively meaningful saliency maps [ 23 , 33 , 24 , 56 ] . However , without human attention as a reference , it is difficult to quantitatively analyze these saliency maps , which motivates our work . There are also methods that change the architecture of the deep RL network by augmenting it with an explicit artificial attention module , so that one can directly access its attention map [ 49 , 80 , 48 ] . Researchers have taken this approach and compared RL agents ’ attention with human attention [ 52 ] . However , these methods do not apply to general deep RL algorithms since they need to modify the original network architectures and retrain the new ones . 3 Method . We now discuss methods for modeling human attention , training RL agents , and extracting attention information from the trained RL agents . We then discuss how we select data for comparing human versus RL attention , and define the comparison metrics . Human attention data and model We use human expert gaze data from Atari-HEAD dataset [ 85 ] . The original game runs continually at 60Hz [ 6 ] , a speed that is challenging even for professional gamers . Human eye movements were rushed and inaccurate at this speed and hence could not serve as a useful reference . In [ 85 ] , however , the human data were collected in a semi-frame-by-frame mode . Without changing the action manipulation or the reward structures of the games , this design mimicked the frame-by-frame processing of RL agents for the human players . The players were allowed enough time to process the visual input , producing higher-quality data with more precise attention allocation and action execution . These advantages led to world expert-level performance in [ 85 ] –about 35x better than the human performance reported in previous deep RL literature [ 46 , 28 ] . To get human saliency maps for the image states generated by RL agents , we need an accurate human attention model . RL agents make many more mistakes than expert humans and can not reach the late stages of the games . This creates a state distribution that does not match human data , which contains mostly good states . Having a human attention model allows us to perform comparisons in states encountered by RL agents only , which is especially important when we later analyze RL agents ’ failure states . Predicting human attention is a challenging task in Atari games . The human gaze is rarely on the player ’ s avatar or contingency regions [ 6 ] so simple object detectors would not work . Human players often select and focus on a few among multiple visually identical objects , and divide their attention if multiple objects are relevant for decision making . Recently , variants of convolution-deconvolution network models have achieved the best results on predicting human attention [ 42 , 83 , 53 , 16 ] . Here we followed their approach and trained gaze models from ground truth human gaze data . Given an image , the models produce a heatmap of the same dimension that predicts human attention distribution . These models have been evaluated and described in detail in previous studies ( e.g. , [ 83 ] ) , so we leave out the details in Appendix 1 . As a control analysis , we include two attention models in addition to the human gaze . The first one captures the motion information , measured by Farneback optical flow between two consecutive images [ 20 ] . The second model captures salient low-level image features , including color , orientation , and intensity ( weighted equally ) , computed by the classic Itti-Koch saliency model [ 32 ] . Implementation details of these models can be found in Appendix 1 . Reinforcement learning agent and attention model As a case study , we use a popular deep RL algorithm named Proximal Policy Optimization ( PPO ) [ 63 ] with default hyperparameters [ 30 ] . For each experimental condition ( discussed later ) , we train 5 models with different random seeds to capture the variance in training and ensure reproducibility . For the Atari gaming environment , we use the basic version that has no frame skipping and no stochasticity in action execution ( NoFrameskip-v4 version ) . In order to capture variance in training and ensure reproducibility , we select six popular Atari games instead of using all Atari games and train 540 agents in total , with the same hyperpareameters except for random seeds and discount factors ( see Section 4 ; 300 GPU days on GeForce GTX 1080/1080 Ti ) . We use a perturbation-based method [ 23 ] to extract attention maps from PPO agents , which has been validated by several subsequent studies [ 25 , 65 ] . The algorithm takes an input image I and applies a Gaussian filter to a pixel location ( i , j ) to blur the image . This manipulation adds spatial uncertainty to the region around and produces a perturbed image Φ ( I , i , j ) . Denote the learned policy as π and the inputs to the final softmax activation as πu ( I ) for image I ( i.e . the last latent representation ) . A saliency score for this pixel ( i , j ) can be defined as how much the blurred image changes the latent representation πu ( I ) in Euclidean space : Sπ ( i , j ) = 1 2 ‖πu ( I ) − πu ( Φ ( I , i , j ) ) ‖2 . ( 1 ) Intuitively , Sπ ( i , j ) describes how much removing information from the region around location ( i , j ) changes the policy [ 23 ] . In other words , a large Sπ ( i , j ) indicates that the information around pixel ( i , j ) is important for the learning agent ’ s decision making . Instead of calculating the score for every pixel , [ 23 ] found that computing a saliency score for pixel i mod 5 and j mod 5 produced good saliency maps at lower computational cost for Atari games . The final saliency map P is normalized as P ( i , j ) = Sπ ( i , j ) ∑ i , j Sπ ( i , j ) . Comparison metrics Next , we compile a set of game images to compute human and RL saliency maps . For each game , we let a trained PPO agent ( with default hyperparameters ) play the game until terminated , and uniformly sampled 100 images from the recorded trajectory . We will refer to this set of images as the standard image set . We then define two metrics for comparing saliency maps when using RL data : Pearson ’ s Correlation Coefficient ( CC ) and Kullback-Leibler Divergence ( KL ) . Let Q denote the human saliency map predicted by the human attention network . CC is between −1 and 1 captures the linear relation between two distributions Q and P : CC ( P , Q ) = σ ( P , Q ) σP × σQ ( 2 ) where σ ( P , Q ) denotes the covariance , and σP and σQ are the standard deviations of P and Q respectively . CC penalizes false positives and false negatives equally . However , we may not want to penalize the RL agent if it attends to regions that the human gaze model is not currently focused on . The human gaze data only reveals the “ overt ” attention , and humans can still pay “ covert ” attention to entities in the working memory [ 54 , 62 ] . In other words , being attended to by the human gaze is a sufficient ( but not necessary ) condition for the features to be important . Thus we need a second metric that penalizes the agent only if it fails to pay attention to human attended regions , or equivalently , a metric that is sensitive to false negatives if we treat human attention as the ground truth . KL is an ideal candidate in this case [ 9 ] : KL ( P , Q ) = ∑ i ∑ j Q ( i , j ) log ( + Q ( i , j ) + P ( i , j ) ) ( 3 ) where is a small regularization constant ( chosen to be 2.2204e-16 [ 9 ] ) and determines how much zero-valued predictions are penalized .

Summary: - This paper presents an empirical study of machine vs. human attention in Deep RL on Atari.  Specifically they examine how the attention of an CNN agent correlates with human attention - They find that human attention and RL attention correlate well and this correlation improves as agent performance improves.  For attention methods like low-level image feature salience, however correlation with RL attention reduces as performance increases, suggesting that RL attends to higher-level features in the image as training progresses. - The examine how the discount factor changes attention, specifically a lower discount factor helps RL learn to attend to similar things as the human experts. -  Finally they show that in failure states, RL attention often deviates from human attention -- although not always.