Learning what and where to attend

1 INTRODUCTION . Attention has become the subject of intensive research within the deep learning community . While biology is sometimes mentioned as a source of inspiration ( Stollenga et al. , 2014 ; Mnih et al. , 2014 ; Cao et al. , 2015 ; You et al. , 2016 ; Chen et al. , 2017 ; Wang et al. , 2017 ; Biparva and Tsotsos , 2017 ) , the attentional mechanisms that have been considered remain limited in comparison to the rich and diverse array of processes used by the human visual system ( see Itti et al. , 2005 , for a review ) . In addition , whereas human attention is controlled by varying task demands , attention networks used in computer vision are solely optimized for object recognition . This means that , unlike infants who can rely on a myriad of visual cues and supervision to learn to focus their attention ( Itti et al. , 2005 ) , DCNs must solve this challenging problem with weak supervisory signals derived from statistical associations between image pixels and class labels . Here , we investigate how explicit human supervision – teaching DCNs what and where to attend – affects their performance and interpretability . 1.1 RELATED WORK . Attention models in human vision Attention has been extensively studied from a computational neuroscience perspective ( see Itti et al. , 2005 , for a review ) . One key motivation for the present work includes the presence of complementary pathways for visual attention , which operate on global versus local features . Global features encode scene layout or “ gist ” , and are hypothesized to capture the contextual information which guides visual processing to task-relevant image locations ( Oliva and Torralba , 2007 ) . This is most similar to the feature-based attention that is used in state-of-the-art networks ( Bell et al. , 2016 ; Wang et al. , 2017 ; Hu et al. , 2017 ) . Local features in attention models are assumed to drive a separate form of attention known as visual saliency ( Itti and Koch , 2001 ) . A widely held belief is that visual saliency provides a task-independent topographical encoding of feature conspicuity in a visual scene . It has been proposed that the rapid convergence of these two pathways acts as an efficient shortcut for filtering clutter from object detection processes ( Torralba et al. , 2006 ) . Attention networks in computer vision An extensive body of work aims at explicitly predicting human eye fixations and/or detecting objects ( see Nguyen et al. , 2018 , for a review ) . In addition , recent research on image categorization has focused on integrating attention modules within end-toend trainable DCNs . These attention modules fall into two broad categories that are conceptually similar to the global and local pathways studied in visual neuroscience . Feature-based attention ( also called “ channel-wise ” attention ) involves learning a task-specific feature modulation that is applied across an entire scene to refine feature maps . Spatial attention is a complementary form of attention which involves learning a spatial mask that enhances/suppresses the activity of computational units inside/outside an attention “ spotlight '' positioned in a scene . This is done according to the spatial location of individual network units independently of their feature tuning . Such mechanisms have been shown to significantly improve performance on visual question answering and captioning ( e.g. , Nam et al. , 2017 ; Patro and Namboodiri , 2018 ) . Here , we combine spatial- and feature-based attention into a single mask that modulates feature representations , similarly to state-of-the-art approaches ( e.g. , Chen et al. , 2017 ; Wang et al. , 2017 ; Biparva and Tsotsos , 2017 ; Park et al. , 2018 ; Jetley et al. , 2018 ) , with a formulation that additionally supports supervision from human annotations . Humans-in-the-loop computer vision A central goal of the present study is to leverage human supervision to co-train an attention network . Previous work has shown that it is possible to augment vision systems with human perceptual judgments on many difficult recognition problems ( e.g. , Vondrick et al. , 2015 ; Shanmuga Vadivel et al. , 2015 ; Kovashka et al. , 2016 ; Deng et al. , 2016 ) . In particular , online games constitute an efficient way to collect high-quality human ground-truth data ( e.g. , von Ahn et al. , 2006 ; Deng et al. , 2016 ; Das et al. , 2016 ; Linsley et al. , 2017 ) . In a two-player game , an image may be gradually revealed to a “ student ” tasked to recognize it , by a remote “ teacher ” who draws “ bubbles ” on screen to unveil specific image parts ( Linsley et al. , 2017 ) . The game-play mechanics introduced by assembling teams of players reduces annotation noise in these games , but also limits their suitability for large-scale data collection . This limitation is aleviated in one-player games ( Deng et al. , 2013 ; 2016 ; Das et al. , 2016 ) , where a player may be asked to sharpen regions of a blurred image to answer questions about it . The game introduced here differs from these earlier studies ( Linsley et al. , 2017 ) in that it has a human player collaborate with a DCN to discover important visual features for recognition at ImageNet scale . Attention datasets Recording eye fixations while viewing a stimulus is a classic way to explore visual attention ( e.g. , Judd et al. , 2009 ) . It is , however , difficult and costly to acquire large-scale eye tracking data , leading researchers to instead track computer mouse movements during taskfree image viewing . Maps collected from individual observers can then be combined into a single “ bottom-up ” saliency map , as is done with the popular Salicon dataset ( Jiang et al. , 2015 ) . Still , Salicon contains saliency maps for 10k images , which is relatively small for training deep networks on object recognition . We instead describe a gamification procedure used to collect nearly a halfmillion “ top-down ” ( task-driven ) attention maps over several months . As we demonstrate through psychophysics , our top-down attention maps also better reflect human recognition strategies than Salicon ( section 2.2 ) , as they are collected while human observers search for visual features that are informative for object categorization . 1.2 CONTRIBUTIONS . Our contributions are three-fold : ( i ) We describe a large-scale online experiment ClickMe.ai used to supplement ImageNet with nearly a half-million top-down attention maps derived from human participants . These maps are validated using human psychophysics and found to be more diagnostic than bottom-up attention maps for rapid visual categorization . ( ii ) As a proof of concept , we describe an extension of the leading squeeze-and-excitation ( SE ) module : the global-and-local attention ( GALA ) module , which combines global contextual guidance with local saliency and substantially improves accuracy on ILSVRC12 . ( iii ) We find that incorporating ClickMe supervision into GALA training leads to an even larger gain in accuracy while also encouraging visual representations that are more interpretable and more similar to those derived from human observers . By supplementing ImageNet with the public release of ClickMe attention maps , we hope to spur interest in the development of network architectures that are not only more robust and accurate , but also more interpretable and consistent with human vision . 2 CLICKME.AI . A large-scale effort is needed to gather sufficient attention annotations for training neural network models . Our starting point for this endeavor is the Clicktionary game that we introduced in ( Linsley et al. , 2017 ) , which pairs online human participants to cooperatively annotate object images . This two-player game was successfully used to collect attention maps for a few hundred images , but we found it impossible to scale beyond that because of the challenge of matching pairs of players . These limitations prompted us to develop ClickMe.ai , an adaptation of Clicktionary into a single-player game that supports large-scale data acquisition . ClickMe consists of rounds of game play where human participants play with DCN partners to recognize images from the ILSVRC12 challenge . Players viewed one object image at a time and were instructed to use the mouse cursor to “ paint ” image parts that were most informative for recognizing its category ( written above the image ) . Once the participant clicked on the image , 14 × 14 pixel bubbles were placed wherever the cursor went until the round ended ( Fig . 1 ) . Having players annotate in this way forced them to carefully monitor their bubbling while also preventing fastidious strategies that would produce sparse salt-and-pepper types of maps . As players bubbled image regions they deemed important for recognition , a DCN tried to recognize a version of the image in which only these bubbled parts were visible . We tried to make the game as entertaining and fast-paced as possible to maximize the number of rounds played by the human players . Hence , we nearly doubled the size of the bubbled regions shown to the DCN ( 21×21 pixels ) to make sure that the objects were recognizable to the DCN within a few seconds of play ( Fig . 1 ) . The reasons for keeping a DCN in the loop ( as opposed to a random timer ) were ( 1 ) to make the game entertaining , and ( 2 ) to discourage human players from bubbling random locations in an image which are potentially unrelated to the object . Indeed , we have incorporated all the images used in Clicktionary and found that ClickMe and Clicktionary maps are strongly correlated ( see Appendix ; ( Linsley et al. , 2017 ) ) . This suggests that replacing one of the human players with a DCN did not bias the collected annotations , but it did allow us to collect human data at a scale suitable for co-training a DCN with human attention annotations.1 A timer controlled the number of points participants received per image , and high-scoring participants were awarded prizes ( see Appendix ) . Points were calculated as the proportion of time left on the timer after the DCN achieved top-5 correct recognition for the image . If the player could not help the DCN recognize the object within 7 seconds , the round ended and no points were awarded . 2.1 GAME STATISTICS . Data collection efforts on ClickMe.ai drew 1,235 participants ( unique user IDs ) who played an average of 380 images each . In total , we recorded over 35M bubbles , producing 472,946 ClickMe maps on 196,499 unique images randomly selected from our preparation of ILSVRC12 ( see Appendix for details ) . All ClickMe maps were used in subsequent experiments , regardless of the ability of the DCN partner to correctly identify them within the time limit of a round . Fig . 2A shows sample ClickMe maps where pixel opacity reflects the likelihood that it was bubbled by participants . ClickMe maps often highlight local image features and emphasize specific object parts over others . For instance , ClickMe maps for animal categories ( Fig . 2A , top row ) are nearly always oriented towards facial components even when these are not prominent ( e.g. , snakes ) . In general , we also found that ClickMe maps for inanimate objects ( Fig . 2A , bottom row ) exhibted a front-oriented bias , with distinguishing parts such as engines , cockpits , and wheels receiving special attention . Additional game information , statistics and ClickMe maps are available as Appendix . 1See Appendix for an extended discussion on why human observers likely adopted strategies for selecting object features relevant to their recognition instead of performing a sensitivity analysis on their DCN partners .

Summary: This paper argues that most recent gains in visual recognition are due to the use of visual attention mechanisms in deep convolutional networks (DCNs). According to the authors; the networks learn where to focus through a weak form of supervision based on image class labels. This paper introduces a data set that complements ImageNet with circa 500,000 human-derived attention maps, obtained through a large-scale online experiment called ClickMe. These attention maps can be used in conjunction with DCNs to add a human-in-the-loop feature that significantly improves accuracy.